How to copy files as fast as possible?

Question

I am running my shell script on machineA which copies the files from machineB and machineC to machineA.

If the file is not there in machineB, then it should be there in machineC for sure. So I will try to copy from machineB first, if it is not there in machineB then I will go to machineC to copy the same files.

In machineB and machineC there will be a folder like this YYYYMMDD inside this folder -

/data/pe_t1_snapshot

So whatever date is the latest date in this format YYYYMMDD inside the above folder - I will pick that folder as the full path from where I need to start copying the files -

so suppose if this is the latest date folder 20140317 inside /data/pe_t1_snapshot then this will be the full path for me -

/data/pe_t1_snapshot/20140317

from where I need to start copying the files in machineB and machineC. I need to copy around 400 files in machineA from machineB and machineC and each file size is 1.5 GB.

Currently I have my below shell script which works fine as I am using scp but somehow it takes ~2 hours to copy the 400 files in machineA which is too long for me I guess. :(

Below is my shell script -

#!/bin/bash

readonly PRIMARY=/export/home/david/dist/primary
readonly SECONDARY=/export/home/david/dist/secondary
readonly FILERS_LOCATION=(machineB machineC)
readonly MEMORY_MAPPED_LOCATION=/data/pe_t1_snapshot
PRIMARY_PARTITION=(0 3 5 7 9) # this will have more file numbers around 200
SECONDARY_PARTITION=(1 2 4 6 8) # this will have more file numbers around 200

dir1=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[0]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
dir2=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[1]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)

echo $dir1
echo $dir2

if [ "$dir1" = "$dir2" ]
then
    # delete all the files first
    find "$PRIMARY" -mindepth 1 -delete
    for el in "${PRIMARY_PARTITION[@]}"
    do
        scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.
    done

    # delete all the files first
    find "$SECONDARY" -mindepth 1 -delete
    for sl in "${SECONDARY_PARTITION[@]}"
    do
        scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/.
    done
fi

I am copying PRIMARY_PARTITION files in PRIMARY folder and SECONDARY_PARTITION files in SECONDARY folder in machineA.

Is there any way to move the files faster in machineA. Can I copy 10 files at a time or 5 files at a time in parallel to speed up this process or any other approach?

NOTE: machineA is running on SSD

UPDATE:-

Parallel Shell Script which I tried, top portion of shell script is same as shown above.

if [ "$dir1" = "$dir2" ] && [ "$length1" -gt 0 ] && [ "$length2" -gt 0 ]
then
    find "$PRIMARY" -mindepth 1 -delete
    for el in "${PRIMARY_PARTITION[@]}"
    do
        (scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.) &
          WAITPID="$WAITPID $!"        
    done

    find "$SECONDARY" -mindepth 1 -delete
    for sl in "${SECONDARY_PARTITION[@]}"
    do
        (scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/.) &
          WAITPID="$WAITPID $!"        
    done
     wait $WAITPID
     echo "All files done copying."
fi

Errors I got with parallel shell script-

channel 24: open failed: administratively prohibited: open failed
channel 25: open failed: administratively prohibited: open failed
channel 26: open failed: administratively prohibited: open failed
channel 28: open failed: administratively prohibited: open failed
channel 30: open failed: administratively prohibited: open failed
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
channel 32: open failed: administratively prohibited: open failed
channel 36: open failed: administratively prohibited: open failed
channel 37: open failed: administratively prohibited: open failed
channel 38: open failed: administratively prohibited: open failed
channel 40: open failed: administratively prohibited: open failed
channel 46: open failed: administratively prohibited: open failed
channel 47: open failed: administratively prohibited: open failed
channel 49: open failed: administratively prohibited: open failed
channel 52: open failed: administratively prohibited: open failed
channel 54: open failed: administratively prohibited: open failed
channel 55: open failed: administratively prohibited: open failed
channel 56: open failed: administratively prohibited: open failed
channel 57: open failed: administratively prohibited: open failed
channel 59: open failed: administratively prohibited: open failed
mux_client_request_session: session request failed: Session open refused by peer
channel 61: open failed: administratively prohibited: open failed
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
channel 64: open failed: administratively prohibited: open failed
mux_client_request_session: session request failed: Session open refused by peer
channel 68: open failed: administratively prohibited: open failed
channel 72: open failed: administratively prohibited: open failed
channel 74: open failed: administratively prohibited: open failed
channel 76: open failed: administratively prohibited: open failed
channel 78: open failed: administratively prohibited: open failed

First, determine where the bottleneck is. If you're already transferring ~100-110MB/s, then you've hit the limits of gigabit ethernet, and you'll have to see if you can get sending compressed data and then decompressing it to be a net gain, or do link aggregation, or move to ten gigabit or infiniband, or physically moving SSD's or entire RAIDsets of SSD's from one machine to the other. — Anti-weakpasswords, Apr 07 '14 at 04:46
What speed is your network? As far as I can tell, the numbers you're giving comes to about 670Mbit/s, so don't expect to gain huge amounts of time on a gigabit network. — Joachim Isaksson, Apr 07 '14 at 04:48
1.5 GB * 400 = 600 GB. 1000BASE-T ethernet can transfer 100 MB/s, so 10 sec / GB, which is 6,000 seconds for 6,000 GB. One hour is 3,600 seconds, and so two hours sounds very close to being as fast as possible. — Joseph Myers, Apr 07 '14 at 04:49
If I am not wrong, we have 10 GIG NIC on machineA but not on machineB and machineC. — john, Apr 07 '14 at 04:51
The files which I am transferring are memory-mapped files which I guess might be already compressed but not sure.. — john, Apr 07 '14 at 04:52
Maybe this is obvious, but have you tried using `scp -C` to enable mutual compression between the SSH endpoints? — Joseph Myers, Apr 07 '14 at 04:53
I already tried using `scp -C` earlier but it didn't helped me at all. And then I decided to use scp with control master so that I can reuse the connection. — john, Apr 07 '14 at 04:55
@JosephMyers: What is link aggregation? I am not sure on this. — john, Apr 07 '14 at 04:56
link aggregation (aka LACP/802.3ad/802.1ax) is a protocol that allows 1000BASE-T ethernet switches to pool the bandwidth of two or more ethernet cables to the same device. So if machine B and C have two or more ethernet ports, you can aggregate them and get 200+ MB/sec throughput from them. A supported switch is needed, and proper network configuration on the switch and machine B and C. — Joseph Myers, Apr 07 '14 at 05:01
By the way, along the lines of aggregation, another thought occurred to me. Have you tried the experiment of simultaneously downloading half of the files from machine B and half of them from machine C? This should cut your time in half even without link aggregation, since your machine A could easily handle the bandwidth of two simultaneous 100 MB/sec connections. — Joseph Myers, Apr 07 '14 at 05:04
@JosephMyers: I am going to give one more shot for `-C`. So this syntax looks like right to you? `scp -p -C -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.` If yes, then I would replicate this to other scp statement — john, Apr 07 '14 at 05:08
Yes, that looks good to me. Also, before trying it, you can run `gzip -v < filename > /dev/null` on a few of the files you are going to be transferring just to see if any of them are compressible or not, rather than doing a two-hour experiment. By default `scp -C` uses gzip compression. — Joseph Myers, Apr 07 '14 at 05:14
Ok. But what this `gzip -v < filename > /dev/null` will tell me that files are not compressible? Any errors or some message by which I can know that files won't be compressed? I will start the run. Now coming back to your last suggestion. How can I download half files from machineB and half files from machineC simultaneously? As I don't know which files will be in machineB and which files in machineC? And you saying download the files in parallel? — john, Apr 07 '14 at 05:16
Yes, download them in parallel. Just make sure that you maintain at least one active download to both B and C at any given point in time, without downloading any duplicates, either. At the very end you might have a few files that are not downloaded in parallel, depending on whether there were more files available on only one of them than on the other. — Joseph Myers, Apr 07 '14 at 05:20
Ok. Coming back to original question. If the files are not compressible then this will tell me `gzip -v < filename > /dev/null` anything? or how would I get to know that files are compressible so I can use `-C`. — john, Apr 07 '14 at 05:22
The `gzip -v` flag tells you the compression level achieved on a file, which allows you to know if it was compressible or not. (It just means "verbose" and is a way of finding out the compression ratio.) — Joseph Myers, Apr 07 '14 at 05:22
I just did on one single file and this was the output printed out on the console - `71.4%`. What do you think now? — john, Apr 07 '14 at 05:23
That means you can very likely achieve a very big speedup from the compression, as long as there is enough CPU time available to your `scp` processes to compress and decompress faster than the available bandwidth. (The bandwidth taken up by that file when compressed would be less than a third of its original size, for example.) — Joseph Myers, Apr 07 '14 at 05:25
However, try doing an experiment with using `scp` to copy just one file with -C and comparing it to the time for just one file without it. Since you mentioned it didn't make a difference earlier, maybe some external process was already compressing the data. So it would be good to know for sure. — Joseph Myers, Apr 07 '14 at 05:27
@JosephMyers: good point, I can do that experiment as well. For now, I have started a run to copy all the files but with `-C` parameter to see again whether this makes any difference or not. After that, I will do that on one single file. In the meantime, coming back to the parallel example. I did tried downloading them in parallel but I got bunch of exceptions on my console related to `mux_client_request_session errors`. Updated the question with parallel shell script which I tried and errors as well. — john, Apr 07 '14 at 05:38
Your parallel script is attempting to download ALL files in parallel. You only need run two parallel processes, one downloading from B and one from C. Each process removes a file name from the master list, puts that file name into its own list, and tries to download it. If downloading is unsuccessful, then it puts that file name back onto the master list. The loop then repeats itself again, making sure not to try to download any file if its name is already in its own list. You simply run two processes which do this in parallel, one each from machine B and C. — Joseph Myers, Apr 07 '14 at 19:03
@JosephMyers: Thanks for suggestion. Can you provide an example on this how would I do that? I am not that much into shell scripting so will be little hard for me to try this out. If you can provide an example then it will be of great help to me. — john, Apr 07 '14 at 19:15
my solution can move file upto 6gb in 1 second https://stackoverflow.com/questions/4743094/how-can-i-move-all-the-files-from-one-folder-to-another-using-the-command-line — pankaj, Aug 13 '20 at 17:30

score 21 · Answer 1 · answered Apr 08 '14 at 15:44

21

you can try this command

rsync

from the

man rsync

you will see that: The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network connection, using an efficient checksum-search algorithm described in the technical report that accompanies this package.

answered Apr 08 '14 at 15:44

oohcode

431
2
6

Just to add - I already have tried `rsync` and somehow rsync was slower as compared to scp with control master. I instrumented both of them.. I user rsync with avz parameter. – john Apr 09 '14 at 02:02
@user2809564 how did you instrument them? – Olimpiu POP Apr 18 '14 at 04:28
@user503413: By copying the same amount of files with rsync shell script as compared to scp shell script.. – john Apr 21 '14 at 04:09

osgx · Answer 2 · 2014-04-14T03:23:11.873

You may try the HPN-SSH (High Performance SSH/SCP) - http://www.psc.edu/index.php/hpn-ssh or http://hpnssh.sourceforge.net/

The HPN-SSH project is the set of patches for OpenSSH (scp is part of it), to better tune various tcp and internal buffers. There is also "none" cipher ("None Cipher Switching") which disables encryption, and this may help you too (if you don't use public networks to send the data).

Both compression and encryption consumes CPU time; and 10 Gbit Ethernet sometimes may be faster to transfer uncompressed file then waiting CPU to compress and encrypt it.

You may profile your setup:

Measure the network bandwidth between machines using iperf or netperf. Compare with the actual network (network cards capabilities, switches). With good setup you should get more than 80-90 percents of declared speed.
Calculate data volume and the time needed to transfer so much data with your network using speed from iperf or netperf. Compare with actual transfer time, is there huge difference?
- If your CPU is fast, data is compressible and network is slow, compressing will help you.
Take a look on top, vmstat, iostat.
- Are there 100% loaded CPU cores (run top and press 1 to see cores)?
- Are there too much interrupts (in) in vmstat 1? What about context switches (cs)?
- What is file reading speed in iostat 1? Are your HDDs are fast enough to read data; to write data on receiver?
You can try to do full-system profiling using perf top or perf record -a. Is there lot of computing by scp, or network stack in Linux? If you can install dtrace or ktap, try to make also off-cpu profiling

score 6 · Answer 3 · answered Apr 17 '14 at 01:21

You have 1.5 GB * 400 = 600 GB of data. Unrelated to the answer I suggest that the machine set up looks incorrect if you need to transfer this amount of data. You probably needed to generate this data at machine A in the first place.

There are 600 GB of data being transferred in 2 hours, that is ~ 85 MB/s transfer rate, which means you probably reached the transfer limits of either your disk drives or (almost) the network. I believe you won't be able to transfer faster with any other command.

If the machines are close to each other, the method of copying that I believe is the fastest is to physically remove the storage from machines B and C, put them in machine A and then locally copy them without transferring via the network. The time for this is the time to move around the storage, plus disk transfer times. I'm afraid, however, the copy won't be much faster than 85 MB/s.

The network transfer command that I believe would be the fastest one is netcat, because it has no overhead related to encryption. Additionally, if the files are not media files, you have to compress them using a compressor that compresses faster than 85 MB/s. I know of lzop and lz4 that are granted to be faster than this rate. So my command line for transfering a single directory would be (BSD netcat syntax):

machine A:

$ nc -l 2000 | lzop -d | tar x

machine B or C (can be executed from machine A with the help of ssh):

$ tar c directory | lzop | nc machineA 2000

Remove the compressor if transfering media files, which are already compressed.

The commands to organize your directory structure are irrelevant in terms of speed, so I didn't bother to write them here, but you can reuse your own code.

This is the fastest method I can think of, but, again, I don't believe this command will be much faster that what you already have.

score 5 · Answer 4 · answered Jul 15 '19 at 12:47

5

You definitely want to give rclone a try. This thing is crazy fast :

sudo rclone sync /usr /home/fred/temp -P -L --transfers 64

Transferred: 17.929G / 17.929 GBytes, 100%, 165.692 MBytes/s, ETA 0s Errors: 75 (retrying may help) Checks: 691078 / 691078, 100% Transferred: 345539 / 345539, 100% Elapsed time: 1m50.8s

This is a local copy from and to a LITEONIT LCS-256 (256GB) SSD.

answered Jul 15 '19 at 12:47

Frédéric N.

51
1
1

https://stackoverflow.com/questions/4743094/how-can-i-move-all-the-files-from-one-folder-to-another-using-the-command-line – pankaj Aug 13 '20 at 17:30

score 0 · Answer 5 · answered Apr 17 '14 at 00:39

rsync optionally compresses its data. That typically makes the transfer go much faster.

You didn't mention SCP, but SCP -C also compresses.

Do note that compression might make the transfer go faster or slower, depending upon the speed of your CPU and of your network link.

Slower links and faster CPU make compression a good idea; faster links and slower CPU make compression a bad idea.

As with any optimization, measure the results in your own environment.

Also I think ftp is another option for you, as my transfer speed test for large files (>10M) FTP work faster then SCP and even rsync (It's depended on file format and compression rate).

Jeff Sheffield · Answer 6 · 2014-04-17T20:02:37.103

0

rsync is a good answer, but if you care about security then you should consider using:

rdist

Some details on the differences between rsync and rdist can be found here: rdist vs rsync and a blog about how to set it up using ssh can be found here: non root remote updating

Finally you could use the infamous tar pipe tar pattern, with a sprinkle of ssh.

tar zcvf - /wwwdata | ssh root@dumpserver.nixcraft.in "cat > /backup/wwwdata.tar.gz"

This example is talked about here: tar copy over secure network

edited Apr 17 '14 at 20:02

answered Apr 17 '14 at 19:44

Jeff Sheffield

5,768
3
25
32

Can you update links to rdist and/or describe it here? – osgx Mar 31 '15 at 00:04

score 0 · Answer 7 · answered Sep 25 '18 at 10:38

The remote doesn't support ssh multiplexing.

To silence the message:

mux_client_request_session: session request failed: Session open refused by peer

Change your ~/.ssh/config file:

Host destination.hostname.com
  ControlMaster no

Host *
  ControlMaster auto
  ControlPersist yes
  ControlPath ~/.ssh/socket-%r@%h:%p

More details and notes can be found here.

How to copy files as fast as possible?

7 Answers7