this post was submitted on 10 Nov 2023
1 points (100.0% liked)

Data Hoarder

24 readers
1 users here now

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time (tm) ). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

founded 10 months ago
MODERATORS
[email protected]
1
rsync Debian to macOS has copied only 1TB in 42 hours(!) (alien.top)
submitted 10 months ago by [email protected] to c/[email protected]
3 comments fedilink hide all child comments
 

Is there a way to speed this up?

sudo rsync -aogW --partial --progress --exclude-from=/etc/restic.excludes  -e 'ssh -p 22' / [email protected]:/Volumes/TD/rsync/beast_2023_11_07_1106

On the Debian side, rsync 3.2.7
On the iMac Pro side, rsync 3.2.7
Wired ethernet, 10GbE, send speed (in Mission Control) varies around 250MB/s to 500MB/s.
There are about 6.7M files to transfer. After ~42 hours, 1.1M have transferred.
This is what the output of the rsync command looks like:

tank/data/sleep_08-30/tank/data/sleep/files/md5/73/0905320c84d53c108436297fbc5f48
         50,429 100%   53.76kB/s    0:00:00 (xfr#82163, to-chk=4655176/6705279)
tank/data/sleep_08-30/tank/data/sleep/files/md5/73/090a02e70e1eff0b4bd613026fe67c
         51,262 100%   54.35kB/s    0:00:00 (xfr#82164, to-chk=4655175/6705279)
tank/data/sleep_08-30/tank/data/sleep/files/md5/73/093b2096cb530739c6f62354f1ce37

Restic, on the other hand, is wicked fast backing up to the same external drive hanging off the iMac (even while rsync is simultaneously running). I'm using rsync because I would like to have a file-by-file backup, as this drive is being backed up to Backblaze, and without a file-by-file backup, I would be faced with restoring the whole restic backup (8TB) if I lost my external drive and needed to recover files.
Thanks for any suggestions!

you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 1 points 10 months ago

--size-only or other checksum options might help.

Back in the day, we used to make a cpio archive on the fly, pipe it into ssh and un-cpio it at the other end, because that was faster than rsync for many files. Not sure why.