2

I'm going to be transferring some very big files over a fast 10Gbits/s link between two NAS servers. Both NAS systems have raid6 based storage and is capable of writing with about 160 MB/s on average. I've been experimenting and it seems that scp is actually faster than rsync. Here are two examples of transferring a 100GB VM-image:

scp -p /volume1/PROXMOX/images/110/vm-110-disk-1.qcow2 root@169.254.66.172:/volume1/PROXMOX/images/110/

rsync -v -p -o -g -r -t -D -l --numeric-ids -e ssh /volume1/PROXMOX/images/110/vm-110-disk-1.qcow2 root@169.254.66.172:/volume1/PROXMOX/images/110/

scp: 10 minutes 52 seconds

rsync: 23 minutes 24 seconds

Rsync was run over ssh in this case. So for huge files like this, there seems to be too much overhead for rsync to build checksums etc. Does anyone have a similar experience or is there something I'm missing here? Are there any secret rsync-flags that could speed it up? I

Community
  • 1
  • 1
Hans Otto Lunde
  • 41
  • 1
  • 4
  • Old - https://superuser.com/questions/109780/how-to-speed-up-rsync - but probably still worth reading, I doubt that either `scp` or `rsync` is materially different these days. – High Performance Mark Jun 08 '20 at 10:49
  • If you have money to spend, there are commercial file-transfer solutions which are much faster than anything that runs through ssh. – Kenster Jun 08 '20 at 12:58
  • @Kenster I'd be interested in what you have in mind. I've not found a commercial alternative to rsync/scp/sftp that's any faster. – BDM Feb 26 '21 at 23:15
  • 1
    [Aspera](https://www.ibm.com/products/aspera) [Signiant](https://www.signiant.com/) – Kenster Feb 27 '21 at 03:01

1 Answers1

1

I have a similar experience, and… not really. I'm not aware of any secret flags to speed it up, except maybe don't use -z (compression). I don't have hard data on that, though, and maybe .qcow files are really compressible, I dunno!

Some old tips on the InterWebs have mentioned using a less CPU-intensive cipher algorithm for SSH such as rc4. It looks like RC4 is no longer supported in current OpenSSH, but somebody did compare some of the current ones, and it's a close race. I can't imagine which cipher you use for rsync-over-SSH making a world of difference.

Some other alternatives that get mentioned are to set up the rsync daemon, which doesn't use encryption at all, after the authentication stage, or to use rsync over HPN-SSH with rsync's -e option. Or even using -e rsh, which I guess is theoretically possible if you feel OK about enabling rlogin at your site.

As much as it saddens me to say this, there seems to be a dearth of libre/open source options out there for matching the throughput that you'll see with something like Aspera, recently bought out by IBM. It is bulky, confusing, commercial software, but once I finally get the transfer going with ascp I get 10× transfer rates over rsync between two hosts at different sites. The ascp CLI also has a -T option to disable encryption entirely, which I'll use if the data I'm downloading is already public, like from NCBI or somebody like that.

Without knowing much about this stuff, I can speculate that Aspera's probably using faster crypto routines, on multiple CPU cores, and maybe a lower overhead transfer protocol such as UDP (like BitTorrent does) to get the data across. I'd love for somebody who knows more to chime in with better insights about that, though.

TheDudeAbides
  • 1,821
  • 1
  • 21
  • 29