Implementing retry for requests in Python

Question

How do I implement a retry count of 5 times, 10 seconds apart when sending a POST request using the requests package. I have found plenty of examples for GET requests, just not post.

This is what I am working with at the moment, sometimes I get a 503 error. I just need to implement a retry if I get a bad response HTTP code.

for x in final_payload:
    post_response = requests.post(url=endpoint, data=json.dumps(x), headers=headers)

#Email me the error
if str(post_response.status_code) not in ["201","200"]:
        email(str(post_response.status_code))

Possible duplicate of [What is the best way to repeatedly execute a function every x seconds in Python?](https://stackoverflow.com/questions/474528/what-is-the-best-way-to-repeatedly-execute-a-function-every-x-seconds-in-python) — Alex W, Mar 06 '18 at 00:01
Retries have nothing specifically to do with GET or POST, so you should easily be able to adapt the examples you found. — John Gordon, Mar 06 '18 at 00:03
I've used this in the past for GET requests from an external API https://pypi.python.org/pypi/retrying — jrjames83, Mar 06 '18 at 00:09

Dhia · Accepted Answer · 2020-02-04T23:03:19.380

43

you can use urllib3.util.retry module in combination with requests to have something as follow:

from urllib3.util.retry import Retry
import requests
from requests.adapters import HTTPAdapter

def retry_session(retries, session=None, backoff_factor=0.3):
    session = session or requests.Session()
    retry = Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        method_whitelist=False,
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    return session

Usage:

session = retry_session(retries=5)
session.post(url=endpoint, data=json.dumps(x), headers=headers)

NB: You can also inherit from Retry class and customize the retry behavior and retry intervals.

edited Feb 04 '20 at 23:03

answered Mar 06 '18 at 00:14

Dhia

10,119
11
58
69

@DhiaTN - I'm a little confused about this. Assuming this I do `res = session.get(...` - will `res` return immediately in any case? or "wait" until there is either and error or a proper response? What happens if the maximum amount of retries is exceeded? – zerohedge Sep 13 '18 at 19:51
you will response anyway, then you have to check the status code to understand if it failed or not – Dhia Sep 14 '18 at 21:21
For me doing the above didn’t work. I used `from requests.packages.urllib3.util.retry import Retry` instead of `from urllib3.util.retry import Retry`. – zerohedge Sep 15 '18 at 10:42
This gives me an error `ValueError: Unable to determine whether fp is closed.` – Leslie Alldridge Mar 27 '22 at 20:32

score 13 · Answer 2 · edited Oct 28 '20 at 19:46

13

I found that the default behaviour of Retries did not apply to POST. To do so required the addition of method_whitelist, e.g. below:

def retry_session(retries=5):
    session = Session()
    retries = Retry(total=retries,
                backoff_factor=0.1,
                status_forcelist=[500, 502, 503, 504],
                method_whitelist=frozenset(['GET', 'POST']))

    session.mount('https://', HTTPAdapter(max_retries=retries))
    session.mount('http://', HTTPAdapter(max_retries=retries))

    return session

edited Oct 28 '20 at 19:46

Stéphane Bruckert

21,706
14
92
130

answered Jan 24 '19 at 22:55

Giles Knap

495
6
15

7

You should also be able to do `method_whitelist=False` to allow retry on POST. – Matthew Moisen Apr 09 '19 at 00:25
3

Is there a way I can add custom logging for each retry? – Sean E. Sep 16 '21 at 12:16

Implementing retry for requests in Python

2 Answers2

Linked