Curl retry mechanism

Question

I have a script I run to deploy two web services and a front-end application. The script calls a method that checks to see if the two back-end services are up and running. If so, deem the deploy successful, otherwise revert.

The two services sometimes take longer than the front-end to start up. Currently, I put in a sleep call to delay the web service check. This allows them time to start up.

I want to remove this sleep and add in a retry mechanism so that if a service is down, just retry the check repeatedly until I get a response.

To check if the device is up, I use curl. I've read that curl has a retry mechanism but I've never used it.

Have any of you solved this problem before? I want to understand the things I must consider when solving it, e.g. do I retry until I get an HTTP 200 response from my service?

Does anyone have any suggestions on how I'd test this? I'd need to find a service that was down.

EDIT: I see that -retry only reacts to transient errors. "Transient error means either: a timeout, an FTP 4xx response code or an HTTP 5xx response code". My service can return a 404 therefore curl's retry is not my solution.

score 160 · Answer 1 · edited Jun 04 '20 at 07:34

160

The following statement will retry 5 times or a maximum of 40 seconds with a connection timeout of 5 seconds, and no exponential backoff policy

curl --connect-timeout 5 \
    --max-time 10 \
    --retry 5 \
    --retry-delay 0 \
    --retry-max-time 40 \
    'http://your_url'


--max-time 10     (how long each retry will wait)
--retry 5         (it will retry 5 times)
--retry-delay 0   (an exponential backoff algorithm)
--retry-max-time  (total time before it's considered failed)

Note that there is also a --retry-connrefused (since curl 7.52.0) that retries even when the connection is refused and --retry-all-errors (since curl 7.71.0) which "is the sledgehammer of retrying".

edited Jun 04 '20 at 07:34

Jens Erat

37,523
16
80
96

answered Mar 18 '17 at 10:50

Dan Ionescu

3,135
1
12
17

2

Why do you suggest turning off exponential back-off? – TheCoder Mar 18 '17 at 10:52
4

Am an not suggesting to set it off, it is an example of usage the default is off, you can set it to whatever you like. – Dan Ionescu Mar 18 '17 at 10:54
2

I tried the following: (curl -o /dev/null --retry 9 -v -L -w "%{http_code}" ${url} and get the following Warning: Transient problem: HTTP error Will retry in 1 seconds. 9 retries. But, there were no retries. – TheCoder Mar 20 '17 at 08:54
2

In case the --retry-connrefused "seems" ignored, for instance in a docker container, try adding the "-4" option which makes curl use IPv4. See here: https://github.com/curl/curl/issues/5080 – sveri Jun 12 '20 at 17:08
1

The command given here works (apart from the 404 issue) but you should also be careful what you do with the output. If you're using -o you'll get the output of the last (hopefully successful) command. If you're just redirecting with > or | then you'll get the combination of whatever the server output as an error message prepended to the data you want. – Richard Wheeldon Dec 08 '20 at 09:22
@TheCoder some REST apis will use 400 series replies for "async request has not completed yet". In those cases, the client is expected to poll the end point until 200. – Aron Dec 08 '21 at 06:13
@RichardWheeldon you can add `--fail` option, which will suppress all output from failed responses, even when redirecting instead of using `-o` – marc.guenther Aug 08 '23 at 13:40

score 18 · Answer 2 · answered Apr 13 '18 at 21:18

Just to clear up any possible confusion for anyone new...

Please see: https://curl.haxx.se/docs/manpage.html

--retry

If a transient error is returned when curl tries to perform a transfer, it will retry this number of times before giving up. Setting the number to 0 makes curl do no retries (which is the default). Transient error means either: a timeout, an FTP 4xx response code or an HTTP 5xx response code.

When curl is about to retry a transfer, it will first wait one second and then for all forthcoming retries it will double the waiting time until it reaches 10 minutes which then will be the delay between the rest of the retries. By using --retry-delay you disable this exponential backoff algorithm. See also --retry-max-time to limit the total time allowed for retries.

If this option is used several times, the last one will be used.

Added in 7.12.3.

It will not retry for FTP 4xx it will retry for FTP 5xx – Amit Sadafule Mar 25 '21 at 10:18 — Amit Sadafule, Mar 25 '21 at 10:18

score 13 · Answer 3 · answered Sep 02 '19 at 13:14

If you use --max-time 10 this will kill your connection before 10 seconds so its not useful when downloading.

Following will retry 300 times with 5sec delay and 30sec connection timeout

curl \
    --connect-timeout 30 \
    --retry 300 \
    --retry-delay 5 \
    URL

Curl retry mechanism

3 Answers3

Linked