1

I'm trying to run rsync over ssh in parallel to transfer files between two machines for evaluation purposes. I wanna see how faster can I get compared to a single rsync process.

I tried these two solutions: https://wiki.ncsa.illinois.edu/display/~wglick/Parallel+Rsync but with no great success. https://gist.github.com/rcoup/5358786 (I couldn't make it work)

Based on the first link I run a command like this: 

ssh HOST "mkdir -p ~/destdir/basefolder"
cd ./basefolder; ls | xargs -n1 -P 4 -I% rsync -arvuz -e ssh % HOST:~/destdir/basefolder/.

and I get the files transfered, but it doesn't seem to work well... In this case, It will run a process for every file and folder in the basefolder, but when it finds a folder, it will transfer everything inside that folder using only 1 process.

I tried to use find -type f, but I got problems because I loose the file hierarchy.

Does anyone how some methods to do what I want? (Use rsync in parallel over ssh while keeping files and folders hierarchy).

Daivid
  • 627
  • 3
  • 12
  • 22

1 Answers1

2

Since you tagged your question 'gnu-parallel' the obvious is to refer you to http://www.gnu.org/software/parallel/man.html#EXAMPLE:-Parallelizing-rsync

cd src-dir; find . -type f -size +100000 | parallel -v ssh fooserver mkdir -p /dest-dir/{//}\;rsync -Havessh {} fooserver:/dest-dir/{}
serverhorror
  • 779
  • 5
  • 21
Ole Tange
  • 31,768
  • 5
  • 86
  • 104
  • Thanks @ole-tange. I tried but I couldn't make it work. /bin/bash -c mkdir /bin/bash -c -p /bin/bash: myserver: command not found mkdir: missing operand Try `mkdir --help' for more information. /bin/bash -c /home/myhome/srcfolder/synced/\{//\}\ /bin/bash: -c: option requires an argument /bin/bash: /home/myhome/srcfolder/synced/{//}: No such file or directory sending incremental file list rsync: link_stat "/home/myhome/497/synced/{}" failed: No such file or directory (2) sent 12 bytes received 12 bytes 16.00 bytes/sec total size is 0 speedup is 0.00 rsync error: some files/attrs were... – Daivid Mar 17 '14 at 16:33
  • @user1891812 are you hit by http://stackoverflow.com/questions/16448887/gnu-parallel-not-working-at-all/ – Ole Tange Mar 17 '14 at 16:42
  • That was the problem. I'll include your answer in my tests. – Daivid Mar 17 '14 at 17:07