How to resume interrupted download automatically in curl?

Question

I'm working with curl in Linux. I'm downloading a part of a file in ftp server (using the -r option), but my connection is not good, it always interrupts. I want to write a script which resume download when I'm connected again.

I've used this command, but it's not working:

until curl -r 666-9999 -C - --retry 999 -o "path/to/file" "ftp:/path/to/remote/file"; do :; done

`--retry` is "supposed" to work, see https://stackoverflow.com/questions/42873285/curl-retry-mechanism — rogerdpack, Aug 21 '23 at 18:42

score 123 · Answer 1 · edited Jul 27 '21 at 08:51

123

curl -L -O your_url

This will download the file.

Now let's say your connection is interrupted;

curl -L -O -C - your_url

This will continue downloading from the last byte downloaded

From the manpage:

Use "-C -" to tell curl to automatically find out where/how to resume the transfer. It then uses the given output/input files to figure that out.

edited Jul 27 '21 at 08:51

li ki

342
3
11

answered Nov 17 '17 at 05:42

HexaCrop

3,863
2
23
50

6

yes, and if you want to specify an outfile with --output, this will work, too. with redirection (`> myfile`), it does not (obviously) – phil294 Aug 15 '18 at 19:35
Startlingly, you can just use `curl -L -O -C - your_url` from the start, though it only runs once... – rogerdpack Aug 21 '23 at 18:38

score 27 · Answer 2 · answered Nov 01 '13 at 14:41

27

wget has been built specifically for this use case. From the man page:

Wget has been designed for robustness over slow or unstable network connections;
if a download fails due to a network problem, it will keep retrying until the
whole file has been retrieved.  If the server supports regetting, it will
instruct the server to continue the download from where it left off.

wget is available for almost all Linux distributions - it probably is already installed on yours. Just use wget to download the file, it will re-establish the network connection until the file is completely transferred.

answered Nov 01 '13 at 14:41

sjaensch

907
6
9

Does wget support partial download of a file (like the -r option in curl) ? – Bili the big Nov 01 '13 at 14:42
4

according to the manpage it doesn't support downloading only part of the file, but the `-c` flag allows continuing a download. – Wayne Werner Nov 01 '13 at 14:44
1

As far as I know it doesn't, it does only support resuming a partially-downloaded file as Wayne said. Why would you need this? wget will take care of downloading the whole file (and only the parts you haven't already downloaded). – sjaensch Nov 01 '13 at 14:51
I want to detect automatically internet disconnection when downloading.. how can i do that ? – Bili the big Nov 01 '13 at 14:52
Well you could parse the wget output or have a separate script that pings the server... – sjaensch Nov 01 '13 at 19:58
14

The question asks about `curl`, not `wget`. – Geremia Dec 06 '17 at 02:32
wget doesn't do smb:// though. Hoping for answer on curl. – Nate Apr 18 '18 at 13:06
20

@Geremia If a suggestion resolves the underlying problem -- robustly downloading a file -- why not to share it? – Denis Kulagin Nov 18 '18 at 12:29
I've found if the request needs to specify the byte range then wget doesn't know how to continue and will retry from the start – Kotosif Feb 20 '21 at 20:33
Don’t forget to give wget the `-c` flag, otherwise you’ll overwrite the file with a new from-scratch attempt. – Anton Strogonoff Sep 01 '22 at 07:28

score 21 · Answer 3 · answered Nov 24 '15 at 17:15

21

You can check the exit code in a while loop and resume until the exit code indicates that the download has succeeded:

export ec=18; while [ $ec -eq 18 ]; do /usr/bin/curl -O -C - "http://www.example.com/a-big-archive.zip"; export ec=$?; done

The example is taken from http://ilovesymposia.com/2013/04/11/automatically-resume-interrupted-downloads-in-osx-with-curl/

answered Nov 24 '15 at 17:15

Claudio Floreani

2,441
28
34

1

in my case `curl` exits with other codes, so I check for success code instead: `export ec=18; while [ $ec -ni 0 ]; do /usr/bin/curl -O -C - "http://www.example.com/a-big-archive.zip"; export ec=$?; done` – oluckyman Nov 10 '19 at 12:02

How to resume interrupted download automatically in curl?

3 Answers3

Linked