4

I am trying to use rsync to transfer some big files between servers.

For some reasons, when the file is big enough (2GB - 4GB), the rsync would hang in the middle, with the exactly same position, i.e., the progress at which it hanged always stick to the same place even if I retried.

If I remove the file from the destination server first, then the rsync would work fine.

This is the command I used:

/usr/bin/rsync --delete -avz --progress --exclude-from=excludes.txt /path/to/src user@server:/path/to/dest

I have tried to add delete-during and delete-delay, all have no luck.

The rsync version is rsync version 3.1.0 protocol version 31

Any advice please? Thanks!

Walty Yeung
  • 3,396
  • 32
  • 33

3 Answers3

1

Eventually I solved the problem by removing compression option: -z

Still don't know why is that so.

Walty Yeung
  • 3,396
  • 32
  • 33
1

I had the same problem (trying to rsync multiple files of up to 500GiB each between my home NAS and a remote server).

In my case the solution (mentioned here) was to add to "/etc/ssh/sshd_config" (on the server to which I was connecting) the following:

ClientAliveInterval 15
ClientAliveCountMax 240
  • "ClientAliveInterval X" will send a kind of message/"probe" every X seconds to check if the client is still alive (default is not to do anything).
  • "ClientAliveCountMax Y" will terminate the connection if after Y-probes there has been no reply.

I guess that the root cause of the problem is that in some cases the compression (and/or block diff) that is performed locally on the server takes so much time that while that's being done the SSH-connection (created by the rsync-program) is automatically dropped while that's still ongoing.

Another workaround (e.g. if "sshd_config" cannot be changed) might be to use with rsync the option "--new-compress" and/or a lower compression level (e.g. "rsync --new-compress --compress-level=1" etc...): in my case the new compression (and diff) algorithm is a lot faster than the old/classical one, therefore the ssh-timeout might not occur than when using its default settings.

SteStack
  • 21
  • 2
  • EDIT: rsync 3.2.0 ( https://download.samba.org/pub/rsync/NEWS#3.2.0 ) added support for zstd compression, xxhash hashing, and more. I just used ""--compress-choice=zstd --compress-level=3 --checksum-choice=xxh3"" and it's A LOT faster than previously. – SteStack Mar 18 '21 at 20:46
  • EDIT: even by using all optimizations and correct ssh settings etc... I still have the same issue with some files => I was able to reliably reproduce the issue by using a copy of my original source&target files => I managed to anonymize the affected source&target files (to then make them publicly available) while still being able to reproduce the issue => I have opened this issue on Github: https://github.com/WayneD/rsync/issues/217 – SteStack Nov 21 '21 at 22:26
0

The problem for me was I thought I had plenty of disk space on a drive but the partition was not using the whole disk and was half the size of what I expected.

So check the size of the space available with lsblk and df -h and make sure the disk you are writing to reports all the space available on the device.

StevieD
  • 6,925
  • 2
  • 25
  • 45