How to implement retry mechanism into Python Requests library?

Question

I would like to add a retry mechanism to Python Requests library, so scripts that are using it will retry for non-fatal errors.

At this moment I do consider three kind of errors to be recoverable:

HTTP return codes 502, 503, 504
host not found (less important now)
request timeout

At the first stage I do want to retry specified 5xx requests every minute.

I want to be able to add this functionality transparently, without having to manually implement recovery for each HTTP call made from inside these scripts or libraries that are using Python Requests.

Does [this](http://stackoverflow.com/a/15431343/1903116) help? — thefourtheye, Apr 24 '14 at 11:23
@thefourtheye: that only applies to transport-level errors; socket timeouts, SSL errors, and the likes. A server return code in the 500 range is not covered. — Martijn Pieters, Apr 24 '14 at 11:31
Does python-requests handle status code 429? https://tools.ietf.org/html/rfc6585 Sadly, most websites send inappropriate codes when rate limiting (like 503 and 404). — Nemo, Mar 31 '15 at 14:48
requests includes a copy of urllib3's Retry class (in *requests.packages.util.retry.Retry*), which will allow granular control, and includes a backoff mechanism for retry. For status-based retry, use parameter: *status_forcelist* which will force specific status code response to be retried according to the strategy chosen. — datashaman, Feb 25 '16 at 17:09
@datashaman I already implemented what would be called a ResilientSession which takes care of this in a transparent way. Look at my implementation from inside JIRA Python library. — sorin, Feb 25 '16 at 17:47

datashaman · Answer 1 · 2022-02-14T16:36:22.260

247

This snippet of code will make all HTTP requests from the same session retry for a total of 5 times, sleeping between retries with an increasing backoff of 0s, 2s, 4s, 8s, 16s (the first retry is done immediately). It will retry on basic connectivity issues (including DNS lookup failures), and HTTP status codes of 502, 503 and 504.

import logging
import requests

from requests.adapters import HTTPAdapter, Retry

logging.basicConfig(level=logging.DEBUG)

s = requests.Session()
retries = Retry(total=5, backoff_factor=1, status_forcelist=[ 502, 503, 504 ])
s.mount('http://', HTTPAdapter(max_retries=retries))

s.get("http://httpstat.us/503")

See Retry class for details.

edited Feb 14 '22 at 16:36

answered Feb 25 '16 at 19:19

datashaman

8,301
3
22
29

i agree @alex.veprik, but i'm biased! :P – datashaman May 24 '18 at 07:35
1

few Qs * Do I need to put the `s.get("http://httpstat.us/503")` in a `try` and `except` block for the retry to happen or it automatically does it behind the scenes. * Assuming it is done behind the scenes, do I still need to put in a `try` block if in case all retries failed ? – Mateen-Hussain Oct 18 '18 at 08:13
2

The Retry class handles that internally. If all retries fail (or retries are disabled) it throws https://urllib3.readthedocs.io/en/latest/reference/index.html#urllib3.exceptions.MaxRetryError which you can catch if you wish. I've updated the link to the Retry class - check there for more details. – datashaman Oct 18 '18 at 08:29
6

First backoff is 0s, not 1s according to documentation. So it goes like 0s, 2s, 4s, 8s, 16s – epokhe Feb 23 '19 at 09:54
9

What is the significance of the first argument to session.mount, `http://` ? What purpose does it serve? – James Wierzba Feb 27 '19 at 18:56
14

@JamesWierzba it's a pattern to attach to the adapter. So in this case all URLs that start with `http://` will use that adapter. You could have a specific adapter for `https://cnn.com` for example. All calls using that session will go via the custom HTTP adapter when accessing `https://cnn.com ` and the default for everything else. – datashaman Feb 28 '19 at 04:39
7

Look for the `method_whitelist` option in Retry class, by default it doesn't retry on http methods except for `['HEAD', 'TRACE', 'GET', 'PUT', 'OPTIONS', 'DELETE']` - these methods are expected to not have side effects – Sujan Adiga Nov 18 '19 at 12:05
7

Set method_whitelist=False to retry for POST API calls as well. E.g. If you are communicating with GraphQL endpoint all calls are POST. By default only ['HEAD', 'TRACE', 'GET', 'PUT', 'OPTIONS', 'DELETE'] are retried as they are deemed safe / idempotent operations. – amirathi Jun 22 '20 at 10:04
1

you can import `Retry` along with `HTTPAdapter`: `from requests.adapters import HTTPAdapter, Retry` – neurino Feb 14 '22 at 15:07
Thanks @amirathi. I was wondering why POST requests weren't retrying. There's a deprecation error now for the method_whitelist parameter. It's replaced by allowed_methods. Example: ```allowed_methods=["POST"]``` – Nathan Smeltzer Aug 10 '23 at 10:40

Mikel Emaldi Manrique · Answer 2 · 2014-05-15T21:41:39.280

28

This is a snippet of code I used to retry for the petitions made with urllib2. Maybe you could use it for your purposes:

retries = 1
success = False
while not success:
    try:
        response = urllib2.urlopen(request)
        success = True
    except Exception as e:
        wait = retries * 30;
        print 'Error! Waiting %s secs and re-trying...' % wait
        sys.stdout.flush()
        time.sleep(wait)
        retries += 1

The waiting time grows incrementally to avoid be banned from server.

edited May 15 '14 at 21:41

answered Apr 24 '14 at 11:24

Mikel Emaldi Manrique

615
6
12

6

I would rather explicitly use 'continue' in a loop and break by default. Just to avoid infinite loop by mistake. – ed22 Jun 28 '20 at 10:20
10

to limit the number of retries, I could use `while not success and retries < 5:` – blackraven May 08 '21 at 10:06

Danil · Answer 3 · 2019-06-24T09:50:44.237

11

Possible solution using retrying package

from retrying import retry
import requests


def retry_if_connection_error(exception):
    """ Specify an exception you need. or just True"""
    #return True
    return isinstance(exception, ConnectionError)

# if exception retry with 2 second wait  
@retry(retry_on_exception=retry_if_connection_error, wait_fixed=2000)
def safe_request(url, **kwargs):
    return requests.get(url, **kwargs)

response = safe_request('test.com')

edited Jun 24 '19 at 09:50

answered Jun 24 '19 at 07:39

Danil

4,781
1
35
50

1

the retrying package above seems unmaintained. I've had success with the more updated fork: https://github.com/jd/tenacity – chaim Feb 20 '23 at 13:12
@chaim will the code provided in the answer still work with a fork? – Danil Feb 20 '23 at 15:30
@Danill Mashkin, not exactly the same code, the updated fork has changed few things, but quite similar. – chaim Feb 22 '23 at 00:56

score 9 · Answer 4 · answered Jun 30 '20 at 15:07

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry


MAX_RETRY = 2
MAX_RETRY_FOR_SESSION = 2
BACK_OFF_FACTOR = 0.3
TIME_BETWEEN_RETRIES = 1000
ERROR_CODES = (500, 502, 504)


def requests_retry_session(retries=MAX_RETRY_FOR_SESSION,
    back_off_factor=BACK_OFF_FACTOR,
    status_force_list=ERROR_CODES, 
    session=None):
       session = session  
       retry = Retry(total=retries, read=retries, connect=retries,
                     backoff_factor=back_off_factor,
                     status_forcelist=status_force_list,
                     method_whitelist=frozenset(['GET', 'POST']))
       adapter = HTTPAdapter(max_retries=retry)
       session.mount('http://', adapter)
       session.mount('https://', adapter)
       return session



class ConfigService:

   def __init__(self):
      self.session = requests_retry_session(session=requests.Session())

   def call_to_api():
      config_url = 'http://localhost:8080/predict/'
      headers = {
        "Content-Type": "application/json",
        "x-api-key": self.x_api_key
      } 
      response = self.session.get(config_url, headers=headers)
      return response

Does the `self.session.get` in `call_to_api()` need to be wrapped in a try block? — SarahJessica, Jul 21 '20 at 09:05

score -1 · Accepted Answer · edited Oct 01 '18 at 08:50

I was able to obtain the desired level of reliability by extending requests.Session class.

Here is the code https://bitbucket.org/bspeakmon/jira-python/src/a7fca855394402f58507ca4056de87ccdbd6a213/jira/resilientsession.py?at=master

EDIT That code was:

from requests import Session
from requests.exceptions import ConnectionError
import logging
import time


class ResilientSession(Session):

    """
    This class is supposed to retry requests that do return temporary errors.

    At this moment it supports: 502, 503, 504
    """

    def __recoverable(self, error, url, request, counter=1):
        if hasattr(error,'status_code'):
            if error.status_code in [502, 503, 504]:
                error = "HTTP %s" % error.status_code
            else:
                return False
        DELAY = 10 * counter
        logging.warn("Got recoverable error [%s] from %s %s, retry #%s in %ss" % (error, request, url, counter, DELAY))
        time.sleep(DELAY)
        return True


    def get(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).get(url, **kwargs)
            except ConnectionError as e:
                r = e.message
            if self.__recoverable(r, url, 'GET', counter):
                continue
            return r

    def post(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).post(url, **kwargs)
            except ConnectionError as e:
                r = e.message
            if self.__recoverable(r, url, 'POST', counter):
                continue
            return r

    def delete(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).delete(url, **kwargs)
            except ConnectionError as e:
                r = e.message
            if self.__recoverable(r, url, 'DELETE', counter):
                continue
            return r

    def put(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).put(url, **kwargs)
            except ConnectionError as e:
                r = e.message

            if self.__recoverable(r, url, 'PUT', counter):
                continue
            return r

    def head(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).head(url, **kwargs)
            except ConnectionError as e:
                r = e.message
            if self.__recoverable(r, url, 'HEAD', counter):
                continue
            return r

    def patch(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).patch(url, **kwargs)
            except ConnectionError as e:
                r = e.message

            if self.__recoverable(r, url, 'PATCH', counter):
                continue
            return r

    def options(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).options(url, **kwargs)
            except ConnectionError as e:
                r = e.message

            if self.__recoverable(r, url, 'OPTIONS', counter):
                continue
            return r

You should copy-paste that code here. You shouldn't answer a question with something that is offsite - link rot. Also, TBH, your code could be a lot shorter. I think restructure it to be a decorator and you'll have a much shorter, less duplicated solution. — datashaman, Feb 26 '16 at 04:54
Here's a basic rewrite of your resilient session to use decorators. Still has some duplication (the methods all have the same pattern), but it works exactly the same and you only have to change the logic in one place: https://gist.github.com/datashaman/fc02882d6be49d0b882f — datashaman, Feb 26 '16 at 05:20
A more succinct version that does the retry on the request method, removing all duplication from the code: https://gist.github.com/datashaman/a517da0ebfe7939c6b83 — datashaman, Feb 26 '16 at 05:52
I recognize that this code is 5 years old, but it's quality is very low. The answer from @datashaman should be accepted so that this answer can receive less attention. — Justin Johnson, Sep 03 '19 at 17:08

score -1 · Answer 6 · answered Jun 30 '20 at 14:53

Method to retry certain logic if some exception has occured at time intervals t1=1 sec, t2=2 sec, t3=4 sec. We can increase/decrease the time interval as well.

MAX_RETRY = 3
retries = 0

try:

    call_to_api() // some business logic goes here.

except Exception as exception:

    retries += 1
    if retries <= MAX_RETRY:
        print("ERROR=Method failed. Retrying ... #%s", retries)
        time.sleep((1 << retries) * 1) // retry happens after time as a exponent of 2
        continue
    else:
        raise Exception(exception)

How to implement retry mechanism into Python Requests library?

6 Answers6

Linked

Related