Data Hoarder

24 readers

1 users here now

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time (tm) ). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

founded 10 months ago

MODERATORS

[email protected]

rsync Debian to macOS has copied only 1TB in 42 hours(!) (alien.top)

submitted 10 months ago by [email protected] to c/[email protected]

3 comments fedilink hide all child comments

Is there a way to speed this up?

sudo rsync -aogW --partial --progress --exclude-from=/etc/restic.excludes  -e 'ssh -p 22' / [email protected]:/Volumes/TD/rsync/beast_2023_11_07_1106

On the Debian side, rsync 3.2.7
On the iMac Pro side, rsync 3.2.7
Wired ethernet, 10GbE, send speed (in Mission Control) varies around 250MB/s to 500MB/s.
There are about 6.7M files to transfer. After ~42 hours, 1.1M have transferred.
This is what the output of the rsync command looks like:

tank/data/sleep_08-30/tank/data/sleep/files/md5/73/0905320c84d53c108436297fbc5f48
         50,429 100%   53.76kB/s    0:00:00 (xfr#82163, to-chk=4655176/6705279)
tank/data/sleep_08-30/tank/data/sleep/files/md5/73/090a02e70e1eff0b4bd613026fe67c
         51,262 100%   54.35kB/s    0:00:00 (xfr#82164, to-chk=4655175/6705279)
tank/data/sleep_08-30/tank/data/sleep/files/md5/73/093b2096cb530739c6f62354f1ce37

Restic, on the other hand, is wicked fast backing up to the same external drive hanging off the iMac (even while rsync is simultaneously running). I'm using rsync because I would like to have a file-by-file backup, as this drive is being backed up to Backblaze, and without a file-by-file backup, I would be faced with restoring the whole restic backup (8TB) if I lost my external drive and needed to recover files.
Thanks for any suggestions!

top 3 comments

sorted by: hot top controversial new old

[–] [email protected] 1 points 10 months ago

Rsync over ssh needs a specific encryption parameter so it's not wasting CPU cycles on compression and encryption.

Better would be to use the rsync server listening on the source, and pull to the target.

Sorry not able to write the exact commands, work is busy right now. Good luck, hope you can find a solution!

[–] [email protected] 1 points 10 months ago

zfssend might be an option too, but setting up zfs on macOS could be a challenge, and I’m not sure if backblaze will back up a drive with that file system. Also don’t know how reliable zfs on macOS is.

[–] [email protected] 1 points 10 months ago

--size-only or other checksum options might help.

Back in the day, we used to make a cpio archive on the fly, pipe it into ssh and un-cpio it at the other end, because that was faster than rsync for many files. Not sure why.