rsync suddenly hanging indefinitely during transfers

Question

For the past few years, I have been using an rsync one-liner to back up important folders on my Mac Mini desktop (OSX 10.9, 2.5 GHz i5, 4 GB RAM) to a FreeNAS box (0.7.2 Sabanda revision 5266, Pentium D 2.66 GHz, 822MiB RAM [reported by the system, I think there's 1 GB in there]). I am running an rsync daemon on the FreeNAS box. Recently, these transfers have been hanging indefinitely. I have done the usual Google-fu and am unable to identify the source of the problem or a solution.

The one-liner is:

rsync -rvOlt --exclude '.DS_Store'                                  \
      --exclude '.com.apple.timemachine.supported'                  \
      --delete /Volumes/Storage/Music/Albums/ 192.168.1.100::albums

I have tried enabling -vvv and --progress, but there is no pattern that I can discern between what hangs and what doesn't. Heck, if I retry, the same file might hang at a different point during the transfer or not at all. A dry run (-n) does not always succeed either. The only "success" I've had is implementing a timeout (--timeout=10) and rerunning the command over and over. Eventually, I creep along, but with no guarantee of success and at a pace that is unacceptable. I've reached a point where I have one file that I can't get past.

The Mac Mini is connected to my router via 5 GHz. The FreeNAS box is wired into that same router on a 100 mbit port. When transfers are actually going, rsync --progress reports 2.5-4 MB/s. According to --progress, a hang is literally just that—no data transfer is occurring as far as I can tell.

I need help with both the diagnostics and the solution.

I don't have an answer for you, but I'm experiencing the same problem. I've been using the same script for a year or so, and suddenly I'm getting partial transfers. If I run the rsync command manually, it seems to hang after an indeterminate amount of time. Version is 3.0.9 on all machines. I'm going to try rolling back to another version to see if that makes a difference. — user3225131, Jan 22 '14 at 20:13
2018, and the problem still persists. Sometimes not even a `kill -9` kills rsync. — Luis A. Florit, Jan 04 '19 at 20:41
I have the same problem in 2020; I've tried almost everything suggested in these answers, to no avail. I keep getting wild swings in reported transfer speed; a file will transfer at 100 MB/s and then the next file transfers at 8kB/s... This is maddening... I think I'm going to give up on rsync and return to using cp. — User12345, Nov 05 '20 at 03:38
I've got a problem that sounds pretty much the same. It almost always happens 32K into the next file, occasionally at 0 bytes. And there's a non-solution: when I hit `Ctrl+Z` and type `fg enter`, the transfer resumes. It repeats for every large file, until the total of all files remaining to be checked/copied falls below about 150-200 MB. — Luc VdV, Jan 21 '21 at 07:46
I've had rsync hang on a file just over 100KB in size. I did try Ctrl-Z and fg, but rsync didn't actually seem to do anything when it resumed. — Astrid_Redfern, Mar 09 '21 at 16:39
Removing the `--remove-source-files` flag was sufficient for me (I was using it in combination with `--dry-run`). — Eric Wiener, Jul 11 '23 at 17:27

score 35 · Answer 1 · answered May 25 '16 at 18:50

I was having the same problem. Removing -v didn't work for me. My use-case is slightly different in that I'm going from source (EXT4) to ExFAT. The issue for me was that rsync was attempting to preserve device files and permissions, which ExFAT doesn't support. I was using the -hrltDvaP switches. The -D and -a switches seemed to be my problem. The -a switch translates to -rlptgoD (no -H,-A,-X). The -p, -g, and -o switches seemed to be my root cause as rsync was barfing on one or all of those during runtime. Removing -a and specifying -Prltvc switches explicitly is working for me.

bkupcmd="nice -n$nicelevel /usr/bin/rsync -Prltvc --exclude-from=/var/tmp/ignorelist "

This (dropping p, g, and o) worked for me also. I was copying from a remote ext4 drive to a local btrfs drive. — EricS, Sep 05 '21 at 20:19
same issue - answer helped me on debian buster copying from a zfs\nix file system to an exfat disk — fredless, Dec 31 '21 at 02:38

score 22 · Answer 2 · answered Jan 08 '20 at 22:37

22

Try using --whole-file/-W. This command disables the rsync delta-transfer algorithm. That is what worked for us (WSL to OSX)

our full sync flags were -avWPle

(e was because we were using ssh, and that has to be the last flag)

answered Jan 08 '20 at 22:37

Gideon Gyabaah

388
2
7

I can't imagine doing that for multiple TB of data on a regular basis, even if it does literally solve the problem! – Matthew DeNardo May 07 '20 at 15:25
1

The W option did the trick for me. Otherwise, it was hanging after few seconds. I am rsync'ing from Windows10 NTFS (rsync 3.1.2) to Linux exFAT usb drive (rsync 3.2.3) – B Abali Jan 13 '22 at 04:07
This didn't really help me, but it *did* cause `rsync` to generate different errors which I suppose is progress... – rinogo Feb 07 '23 at 01:36

score 21 · Answer 3 · answered Apr 19 '14 at 09:17

21

I've been running into the same thing again and again and it seems to help if you drop the -v option (which is annoying if you need that output).

answered Apr 19 '14 at 09:17

guest

219
2
3

1

What is mind boggling is that (at least on OS X) there is a "--timeout" option, but no "--tries" option! – Michael Oct 17 '16 at 04:48
1

also to add, if you drop the -v option, how do you know if it hangs? – Michael Mar 16 '17 at 21:53
1

@Michael you can use strace to watch the calls made by rsync: `strace -f -e 'trace=!read' rsync ...` – however, removing `-v` did not solve my issue – Larry Williamson May 31 '17 at 14:57
7

@Michael add `--progress` if you drop `-v`. it will help to know if hanging – Jossef Harush Kadouri Aug 08 '17 at 11:17
4

I still get hangs with --progress and no -v – openCivilisation Feb 16 '20 at 05:13

score 15 · Answer 4 · answered May 20 '14 at 02:27

This happened to me when the remote device ran out of space. The error wouldn't show when --verbose option was used; turning that off yielded some STDERR output that explained that the remote device was out of space. When I freed some space, I was able to run rsync again with --verbose and everything went fine.

score 6 · Answer 5 · edited Nov 02 '18 at 11:10

6

Just had a similar problem while doing rsync from harddisk to a FAT32 USB drive. rsync froze already in less than a second in my case and did not react at all after that ... left it with CTRL+C.

Found out that the problem was a combination of usage of hardlinks on the harddisk and having FAT32 filesystem on the USB drive, which does not support hardlinks.

Formatting the USB drive with ext4 solved the problem for me.

edited Nov 02 '18 at 11:10

Community

1
1

answered Nov 18 '17 at 21:33

Alex

1,602
20
33

Yes that seem to have done it for me. Thanks !! – Akshay Mathur Dec 29 '20 at 14:32

score 6 · Answer 6 · answered May 02 '19 at 20:58

In my situation rsync was not actually failing.

I have regular server backups which transfers large files over 500GB+ and have --append-verify or --checkusm over ssh parameters specified.

What I have found upon analysis is that once the client side completes it's file checks then the server side checks start. Which means while the server is doing it's checks the client side will appear hanged and frozen - run htop on the server to rsync working away.

This is likely a non issue if rsync is run in deamon mode on the server and using the rsync protocol instead of ssh for transfers.

On related note, this very LONG wait would trigger SSH timeout and a rsync: connection unexpectedly closed (254 bytes received so far) [sender] error message, sollution is to add ClientAliveInterval 120 and ClientAliveCountMax 720 to /etc/ssh/sshd_config.

score 2 · Answer 7 · answered Jun 25 '20 at 17:34

I've seen this quite often on 3.0.9 on a directory with hardlinks, but it also happened on 3.1.3.

There is a nice analysis in Debian bug 820916: when its internal sockets are congested with errors, rsync could go into a deadlock.

This might have been fixed in a 3.2 release just a few days ago (Jun 2020):

Avoid a hang when an overabundance of messages clogs up all the I/O buffers.

The only good workaround I can think of is, if the problem is not persistent, then put timeout in front of it: timeout rsync <args> <source> <destination>, then retry. If it is persistent for you, you're the lucky one who can debug it :D

score 1 · Answer 8 · edited Mar 02 '19 at 17:24

1

It also happens when the user on target machine has not write permissions on target folder.
You can try giving write permission to others target folder:

sudo chmod -R o+w /path/to/target-folder

edited Mar 02 '19 at 17:24

Eddie C.

918
10
16

answered May 31 '18 at 19:28

Fernando Campos

41
4

tekati · Answer 9 · 2021-10-04T14:56:00.937

In my case, it was the IPC (Intrusion Protection Component) in our firewall. It sees all the TCP SYN packets as a flood attack and kills the connection. I left a rsync over NFS session open and turned off the IPC for the servers firewall rule and it starting working again right away.

rsync -ravh /source /destination

When it happened I was not able to kill the rsync session. It locked up the NFS mount and I would have to reboot the client machine to get it to work again. The strange thing is it would copy some files over then all of a sudden stop. It always seemed to stop on the same file. So I was looking for file issues, permission issues, TCP offloading issues, tried removing the -v in the rsync call. If you are having this issue at least in my case it even happened with a simple.

cp -rp /source /destination

So I knew then to start looking at other factors. So if you have any sort of intrusion protection on a firewall or router between the servers you can try turning that off temporarily to see if it solves your issue as well.

score 0 · Answer 10 · answered Sep 02 '17 at 19:36

Most likely not "your" problem, but I stumbled upon this question when I was researching a similar behavior:

I'm observing "hanging" when the target site has too much io load. e.G. on one of my small business servers, when someone is resyncing his IMAP account and downloading large batchs of data and a backup job runs that writes his data.

In this situation I notice a steep drop in performance for rsync. Noticeable in a high load value in top on the target machine, even though CPU and Mem are fine.

Waiting for the process to finish has helped every time or interrupting and attempting the rsync at a later time again.

score 0 · Answer 11 · answered Apr 05 '18 at 20:03

0

I was having the same problem and it was because I was running out of memory during the rsync. Created a swap file and problem solved.

answered Apr 05 '18 at 20:03

meammeiam

9
1

score 0 · Answer 12 · answered Jul 24 '19 at 16:18

0

Had rsync hanging issue on Ubuntu 16. None of the options above helped. The problem was in the source drive (external SSD) which suddenly became faulty. I tried several disk checks, but all of them stuck. Ended up rebooting the system and disk suddenly became accessible again.

answered Jul 24 '19 at 16:18

apatsekin

1,524
10
12

16.04 LTS or 16.10? – Astrid_Redfern Mar 09 '21 at 16:48

holger ohmacht · Answer 13 · 2020-01-04T00:01:52.147

Holger Ohmacht aka h8ohmh / 8ohmh:

The problem lies in the filesystem buffer / usage of the interworking of harddisk/hw so far as I could investigate.

Temporal solution for local drives (eg. USB3<->HD) : A script which is polling the changing disk space. If no changing free disk space then rsync is stalled and has to be restarted

cmd="rsync -aW --progress --stats --preallocate --super \ <here your source dir> \ <here your dest dir>" eval "$cmd" & rm ./ndf.txt rm ./odf.txt while [[ 0 == 0 ]]; do df > ./ndf.txt cmp ./odf.txt ./ndf.txt res="$?" echo "$res" if [[ $res == 0 ]]; then echo "###########################################" ls -al "./ndf.txt" ls -al "./odf.txt" killall rsync eval "$cmd" & else cp ./ndf.txt ./odf.txt fi sleep 60 done Change <source dir> etc to your paths!

In my case it is always stalling by usage of rsync's --preallocate option (normally because of better disk performance and rescueing continuous blocks), so as long as the disk and filesystem drivers not reworked there just this solution

score 0 · Answer 14 · edited May 04 '23 at 22:49

I had a shell script that synced one machine to another using:

rsync -avzu --blocking-io -e "ssh" (long list of --excludes) homedirname $user@$host:/Users

and this worked beautifully for many years for two machines on local ethernet. I tried it from one Ventura machine to another Ventura machine. It transferred a few files and hung.

The rsync I am using is rsync version 3.2.7 protocol version 31 installed by Homebrew, on both machines.

Tried various voodoo, e.g. -W, no joy. then I took out --blocking-io and whee, it went just fine.

rsync suddenly hanging indefinitely during transfers

14 Answers14

Linked