336

The Python requests module is simple and elegant but one thing bugs me. It is possible to get a requests.exception.ConnectionError with a message like:

Max retries exceeded with url: ...

This implies that requests can attempt to access the data several times. But there is not a single mention of this possibility anywhere in the docs. Looking at the source code I didn't find any place where I could alter the default (presumably 0) value.

So is it possible to somehow set the maximum number of retries for requests?

Brad Koch
  • 19,267
  • 19
  • 110
  • 137
Kirill Zaitsev
  • 4,511
  • 3
  • 21
  • 29
  • 12
    Any update on this with requests at 2.x? Would love a requests.get(url, max_retries=num_max_retries)) implementation. – paragbaxi Jan 27 '14 at 01:33
  • 23
    @paragbaxi: and even better a `requests.get(url, max_retries=num_max_retries, dely_between_retries=3))` – WoJ Feb 27 '15 at 21:26
  • 6
    @WoJ I took your examples and made it reality ;) in `just.get` and `just.post` in https://github.com/kootenpv/just – PascalVKooten Dec 05 '16 at 21:47
  • 6
    Useful article about retries with Requests: https://www.peterbe.com/plog/best-practice-with-retries-with-requests – Gokul Apr 26 '19 at 08:07

6 Answers6

467

This will not only change the max_retries but also enable a backoff strategy which makes requests to all http:// addresses sleep for a period of time before retrying (to a total of 5 times):

import requests

from requests.adapters import HTTPAdapter, Retry

s = requests.Session()

retries = Retry(total=5,
                backoff_factor=0.1,
                status_forcelist=[ 500, 502, 503, 504 ])

s.mount('http://', HTTPAdapter(max_retries=retries))

s.get('http://httpstat.us/500')

As per documentation for Retry: if the backoff_factor is 0.1, then sleep() will sleep for [0.05s, 0.1s, 0.2s, 0.4s, ...] between retries. It will also force a retry if the status code returned is 500, 502, 503 or 504.

Various other options to Retry allow for more granular control:

  • total – Total number of retries to allow.
  • connect – How many connection-related errors to retry on.
  • read – How many times to retry on read errors.
  • redirect – How many redirects to perform.
  • method_whitelist – Set of uppercased HTTP method verbs that we should retry on.
  • status_forcelist – A set of HTTP status codes that we should force a retry on.
  • backoff_factor – A backoff factor to apply between attempts.
  • raise_on_redirect – Whether, if the number of redirects is exhausted, to raise a MaxRetryError, or to return a response with a response code in the 3xx range.
  • raise_on_status – Similar meaning to raise_on_redirect: whether we should raise an exception, or return a response, if status falls in status_forcelist range and retries have been exhausted.

NB: raise_on_status is relatively new, and has not made it into a release of urllib3 or requests yet. The raise_on_status keyword argument appears to have made it into the standard library at most in python version 3.6.

To make requests retry on specific HTTP status codes, use status_forcelist. For example, status_forcelist=[503] will retry on status code 503 (service unavailable).

By default, the retry only fires for these conditions:

  • Could not get a connection from the pool.
  • TimeoutError
  • HTTPException raised (from http.client in Python 3 else httplib). This seems to be low-level HTTP exceptions, like URL or protocol not formed correctly.
  • SocketError
  • ProtocolError

Notice that these are all exceptions that prevent a regular HTTP response from being received. If any regular response is generated, no retry is done. Without using the status_forcelist, even a response with status 500 will not be retried.

To make it behave in a manner which is more intuitive for working with a remote API or web server, I would use the above code snippet, which forces retries on statuses 500, 502, 503 and 504, all of which are not uncommon on the web and (possibly) recoverable given a big enough backoff period.

datashaman
  • 8,301
  • 3
  • 22
  • 29
  • 1
    I am trying to implement your logic, but I dont know if it is working because the log just show one request even the res status is 503. How I can know if the retry is working? See the code: pastebin.com/rty4bKTw – Danilo Oliveira Feb 24 '16 at 20:55
  • 1
    The attached code works as expected. The trick is the *status_forcelist* parameter. This tells the urllib3 package to retry specific status codes. Code: http://pastebin.com/k2bFbH7Z – datashaman Feb 25 '16 at 16:54
  • 1
    urllib3 does not (and should not) think that status 503 is an exception (by default). – datashaman Feb 25 '16 at 17:15
  • Does this work with non-session actions (e.g. requests.post() ?) – Connor Aug 20 '18 at 21:53
  • 1
    @Connor no, the adapter is attached to the session. – datashaman Aug 30 '18 at 08:10
  • 2
    urlib3.Retry is no longer part of requests. this have to imported directly. Suggested edit – user2390183 Apr 16 '19 at 10:25
  • @datashaman total=5 meaning for every not successful request it will request max for 5 times. Is that right? – KT8 Nov 19 '19 at 06:21
  • Docs say: Set to 0 to fail on the first retry. So I'd assume that 1 would fail on the second retry, and so on. So `total=5` would fail on the 6th retry, probably before it attempts it. – datashaman Nov 20 '19 at 07:02
  • How to use this mechanism based on response text? For example for those APIs which return 200 status code along with message "Please retry later." Tried custom Session() with no luck: https://pastebin.com/tJumKGKN – PATAPOsha Feb 03 '20 at 11:13
  • How to retry connections when the server closes with no response? I.e. when the exception is `requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))` – Jakub Klinkovský Apr 01 '20 at 13:28
  • Thanks for the fantastic answer. When you mention in the comments that this does not work for requests.post() - why is it so? (it works when I tried so I am confused) – WoJ Jun 25 '20 at 16:42
  • @WoJ the original question was would it work with `requests.post`, which it won't. It will work with `s.post`, which is the session which has the adapter mounted to it. `requests.post` does not have an adapter mounted. – datashaman Jun 30 '20 at 04:00
  • @JakubKlinkovský this does not handle non-HTTP response errors. you'd have to wrap the call to this in a try/except which retries a few times to handle connection errors. – datashaman Jun 30 '20 at 04:03
  • allowed_methods=frozenset(['GET', 'POST']) for post – Sion C Oct 01 '22 at 17:50
  • @datashaman , Can you guide me in this question? https://stackoverflow.com/q/74000302/8519380 – Sardar Oct 09 '22 at 04:31
226

It is the underlying urllib3 library that does the retrying. To set a different maximum retry count, use alternative transport adapters:

from requests.adapters import HTTPAdapter

s = requests.Session()
s.mount('http://stackoverflow.com', HTTPAdapter(max_retries=5))

The max_retries argument takes an integer or a Retry() object; the latter gives you fine-grained control over what kinds of failures are retried (an integer value is turned into a Retry() instance which only handles connection failures; errors after a connection is made are by default not handled as these could lead to side-effects).


Old answer, predating the release of requests 1.2.1:

The requests library doesn't really make this configurable, nor does it intend to (see this pull request). Currently (requests 1.1), the retries count is set to 0. If you really want to set it to a higher value, you'll have to set this globally:

import requests

requests.adapters.DEFAULT_RETRIES = 5

This constant is not documented; use it at your own peril as future releases could change how this is handled.

Update: and this did change; in version 1.2.1 the option to set the max_retries parameter on the HTTPAdapter() class was added, so that now you have to use alternative transport adapters, see above. The monkey-patch approach no longer works, unless you also patch the HTTPAdapter.__init__() defaults (very much not recommended).

Boris Verkhovskiy
  • 14,854
  • 11
  • 100
  • 103
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • 1
    @user136036: yes, adapters are looked up by longest prefix match; if you want this to apply to *all* urls then `http://` and `https://` are the minimal prefixes to use, see the documentation the answer links to. – Martijn Pieters Apr 26 '15 at 21:00
  • 4
    Note that `HTTPAdapter(max_retries=5)` will only work for certain scenario. From [requests doc](http://docs.python-requests.org/en/latest/api/?highlight=max_retries#requests.adapters.HTTPAdapter), `Note, this applies only to failed DNS lookups, socket connections and connection timeouts, never to requests where data has made it to the server. By default, Requests does not retry failed connections.` To force retry for any status codes, see @datashaman's answer below. – Steven Xu Sep 26 '17 at 03:45
  • @StevenXu: yes, you can configure `Retry()` to alter what failure scenarios are retried. – Martijn Pieters Sep 26 '17 at 06:50
68

Be careful, Martijn Pieters's answer isn't suitable for version 1.2.1+. You can't set it globally without patching the library.

You can do this instead:

import requests
from requests.adapters import HTTPAdapter

s = requests.Session()
s.mount('http://www.github.com', HTTPAdapter(max_retries=5))
s.mount('https://www.github.com', HTTPAdapter(max_retries=5))
David Dehghan
  • 22,159
  • 10
  • 107
  • 95
gizmondo
  • 880
  • 6
  • 13
37

After struggling a bit with some of the answers here, I found a library called backoff that worked better for my situation. A basic example:

import backoff

@backoff.on_exception(
    backoff.expo,
    requests.exceptions.RequestException,
    max_tries=5,
    giveup=lambda e: e.response is not None and e.response.status_code < 500
)
def publish(self, data):
    r = requests.post(url, timeout=10, json=data)
    r.raise_for_status()

I'd still recommend giving the library's native functionality a shot, but if you run into any problems or need broader control, backoff is an option.

Brad Koch
  • 19,267
  • 19
  • 110
  • 137
11

You can use the requests library to accomplish all in one go. The following code will retry 3 times if you receive 429,500,502,503 or 504 status code, each time with a longer delay set through "backoff_factor". See https://findwork.dev/blog/advanced-usage-python-requests-timeouts-retries-hooks/ for a nice tutorial.

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

retry_strategy = Retry(
    total=3,
    backoff_factor=1,
    status_forcelist=[429, 500, 502, 503, 504],
    method_whitelist=["HEAD", "GET", "OPTIONS"]
)
adapter = HTTPAdapter(max_retries=retry_strategy)
http = requests.Session()
http.mount("https://", adapter)
http.mount("http://", adapter)

response = http.get("https://en.wikipedia.org/w/api.php")
Moses
  • 209
  • 3
  • 7
6

A cleaner way to gain higher control might be to package the retry stuff into a function and make that function retriable using a decorator and whitelist the exceptions.

I have created the same here: http://www.praddy.in/retry-decorator-whitelisted-exceptions/

Reproducing the code in that link :

def retry(exceptions, delay=0, times=2):
"""
A decorator for retrying a function call with a specified delay in case of a set of exceptions

Parameter List
-------------
:param exceptions:  A tuple of all exceptions that need to be caught for retry
                                    e.g. retry(exception_list = (Timeout, Readtimeout))
:param delay: Amount of delay (seconds) needed between successive retries.
:param times: no of times the function should be retried


"""
def outer_wrapper(function):
    @functools.wraps(function)
    def inner_wrapper(*args, **kwargs):
        final_excep = None  
        for counter in xrange(times):
            if counter > 0:
                time.sleep(delay)
            final_excep = None
            try:
                value = function(*args, **kwargs)
                return value
            except (exceptions) as e:
                final_excep = e
                pass #or log it

        if final_excep is not None:
            raise final_excep
    return inner_wrapper

return outer_wrapper

@retry(exceptions=(TimeoutError, ConnectTimeoutError), delay=0, times=3)
def call_api():
praddy
  • 169
  • 1
  • 11