165

I'm using the scp shell command to copy huge folder of files.

But at some point in time I had to kill the running command (by Ctrl+C or kill).

To my understanding scp copies files sequentially, so there should be only one partially copied file.

How can same scp command be resumed to not overwrite successfully copied files and to properly handle partially copied files?

P.S. I know I can do this kind of stuff in rsync, but scp is faster for me for some reason and I use it instead.

Adam Smooch
  • 1,167
  • 1
  • 12
  • 27
Bohdan
  • 16,531
  • 16
  • 74
  • 68
  • 4
    I suspect rsync is slower because it spends some extra time doing things that permit you to restart a transfer. You might consider just using rsync to finish the partial transfer. – Keith Thompson Oct 16 '14 at 18:22
  • 2
    If you wanna be clever, you can always try `dd` with a proper `skip` variable, and pipe to `ssh` :) – Mark Nunberg Oct 16 '14 at 18:29
  • 1
    This is off-topic for SO. It belongs on SuperUser, and indeed [has been asked there](https://superuser.com/questions/561658/how-can-i-resume-a-large-scp-file-transfer-when-using-port-forwarding). – Dan Dascalescu Dec 03 '19 at 16:25
  • rsync uses ssh by default under the covers `Typically, rsync is configured to use ssh by default` so...it should be about as fast (though it does a directory listing first, which may give a slower start) – rogerdpack Aug 27 '20 at 15:52

7 Answers7

299

You should use rsync over ssh

rsync -P -e ssh remoteuser@remotehost:/remote/path /local/path

The key option is -P, which is the same as --partial --progress

By default, rsync will delete any partially transferred file if the transfer is interrupted. In some circumstances it is more desirable to keep partially transferred files. Using the --partial option tells rsync to keep the partial file which should make a subsequent transfer of the rest of the file much faster.

Other options, such -a (for archive mode), and -z (to enable compression) can also be used.

The manual: https://download.samba.org/pub/rsync/rsync.html

rogerdpack
  • 62,887
  • 36
  • 269
  • 388
jordi
  • 3,391
  • 2
  • 14
  • 14
  • 86
    Just wanted to note that rsync will resume a batch copy started with scp just fine. – Craig Anderson Nov 04 '16 at 15:54
  • 19
    You'll get "skipping directory ." error, unless you also specify `-r`. Source: http://lucasb.eyer.be/snips/rsync-skipping-directory.html – Artur Czajka Feb 12 '17 at 11:15
  • 5
    Worked for me, but I added --append too.... rsync -P --append -e ssh remoteuser@remotehost:/remote/path /local/path – cnd Oct 10 '18 at 12:51
  • 2
    @CraigAnderson that is the important point that has been left out. The answer sounds like it is suggesting the download should have started with rsync to begin with. – Willa Jun 25 '19 at 19:12
  • 2
    The options given in the answer did not work for me, but this did : `rsync -P -rsh remoteuser@remotehost:/remote/path /local/path` – Bunny Rabbit Jul 20 '19 at 11:57
  • 3
    You can add options to the ssh commad. I was able to resume an scp copy through a jumpbox with: `rsync -r -P -e 'ssh -J jumpuser@jumphost' /local/path remoteuser@remotehost:/remote/path` – xvan Nov 11 '19 at 22:37
  • Sorry, but I can't seem to get this to work for me. My ssh is on port 2222, and I'm using an RSA key to access it. I've tried a few different permutations of the following but keep getting "Permission denied": `rsync -r -P -e 'ssh -p 2222 -i ~/.ssh/id_rsa' remoteuser@remotehost:/remote/path /local/path`. Anything I should check or change to get this to work? – jktravis Feb 22 '20 at 21:28
  • I googled how to rsync over ssh. but people said "use scp, not rsync.. it is not possible" ok now I googled how to resume scp transfers, now I found the solution for rsync via ssh -.- – MilMike Jul 04 '20 at 10:07
  • The `-e ssh` shouldn't be necessary as rsync uses SSH by default...Sadly there's no -P option in BSD/OS ... – rogerdpack Aug 27 '20 at 16:23
  • Symobic links were not copied. Shows `not a regular file`. Any way to copy that also? – vineeshvs Dec 09 '20 at 09:20
  • What if I am on a server I can't `sudo` in to install `rsync`? Say, a computing cluster? – Eduardo Pignatelli Jun 29 '21 at 16:23
  • I'm on a mac and when I ran with `--rsh` I get an error "rsync: Failed to exec shh: No such file or directory (2)" I had to provide the full path of ssh for some reason. `rsync --verbose -P --recursive --rsh $(which shh) ...` – Jerinaw May 03 '22 at 03:09
  • It looks like `-e ssh` is not needed. – baptx Oct 14 '22 at 19:34
54

An alternative to rsync:

Use sftp with option -r (recursively copy entire directories) and option -a of sftp's get command "resume partial transfers of existing files."

Prerequisite: Your sftp implementation has already a get with -a option.

Example:

Copy directory /foo/bar from remote server to your local current directory. Directory bar will be created in your local current directory.

echo "get -a /foo/bar" | sftp -r user@remote_server
rogerdpack
  • 62,887
  • 36
  • 269
  • 388
Cyrus
  • 84,225
  • 14
  • 89
  • 153
  • 12
    Your solution is great because my server doesn't allow login in ssh, so rsync doesn't work. I needed to upload instead of download so here is my command: `echo "put -a foo/bar" | sftp -r server:/foo` – Carlos Rafael Ramirez Mar 30 '17 at 21:56
  • 3
    This seems much viable option for resuming partially downloaded files, because rsync does not seem to be much ssh friendly (with non-default ports). – Mehdi Haghgoo Dec 20 '18 at 07:39
  • 3
    This also works on Windows, no rsync needed – Anton K Jul 03 '21 at 23:10
  • 2
    @codezombie try rsync -P -e "ssh -p 2022" remoteuser@remotehost:/remote/path /local/path to access ssh over port 2022, etc. you can also pass any other ssh option like this. – Pozzo-Balbi Jul 10 '21 at 23:40
27

Since OpenSSH 6.3, you can use reget command in sftp.

It has the same syntax as the get, except that it starts a transfer from the end of an existing local file.

echo "reget /file/path" | sftp -r user@server_name

The same effect has -a switch to the get command or global command-line -a switch of sftp.

Martin Prikryl
  • 188,800
  • 56
  • 490
  • 992
10

Another possibility is to try to salvage the scp you've already started when it stalls.

ctrl+z to background and stop it, then ssh over to the receiving server and login, then exit. Now fg the scp process and watch it resume from 'stalled'!

Puckfist
  • 109
  • 1
  • 2
7

When rsync stalls as well after couple of seconds when initially running fine I ended up with the following brute force solution to start and stop an re-start the download every 60s:

cat run_me.sh
#!/bin/bash
while [ 1 ]
do
  rsync --partial --progress --rsh=ssh user@host:/path/file.tgz file.tgz &
  TASK_PID=$!
  sleep 60
  kill $TASK_PID
  sleep 2
done
rogerdpack
  • 62,887
  • 36
  • 269
  • 388
remigiusz boguszewicz
  • 428
  • 4
  • 7
1

You can make use of the -rsh and -P options of rsync. -P is for partial download and -rsh indicates transfer is over ssh procotol.

The complete command would be : rsync -P -rsh remoteuser@remotehost:/remote/path /local/path

Bunny Rabbit
  • 8,213
  • 16
  • 66
  • 106
  • Same as jordi's answer? – rogerdpack Aug 27 '20 at 16:27
  • 2
    Which version of rsync is this for? My relatively modern version (rsync version 3.1.2 protocol version 31), doesn't have a `-rsh` option with a single dash, only `--rsh=COMMAND` which the manual says is a synonym for `-e COMMAND` – Michael Firth May 29 '21 at 11:35
-6

I got the same issue yesterday, transfering a huge sql dump over via scp, I got lucky with wget --continue the_url

That blog post explains it quite well http://www.cyberciti.biz/tips/wget-resume-broken-download.html basically:

wget --continue url
rogerdpack
  • 62,887
  • 36
  • 269
  • 388
Arnaud Bouchot
  • 1,885
  • 1
  • 21
  • 19