3

I'm trying to make requests retry if the response status code is not 200, so I wrote this function:

    def _get_resource(url, max_retries=0, delay=0):
         number_retries = 0
         response = requests.get(url)
         while response.status_code != 200 and number_retries < max_retries:
            time.sleep(delay)
            response = requests.get(url)
            number_retries += 1
        response.raise_for_status()
        return response.content

Any suggestions to improve it?

Walid Saad
  • 951
  • 1
  • 8
  • 14
  • At minimum, you'll need to define `self`. It's undefined in this sample code. – Jonathan Eunice Jan 10 '16 at 23:14
  • I edited the code to remove it – Walid Saad Jan 10 '16 at 23:16
  • If you don't want a delay between the retries, there is built-in support by specifying your own transport adapter: http://stackoverflow.com/a/18190415/125629 – John Keyes Jan 10 '16 at 23:19
  • Retry code can use a constant delay, but often it makes more sense to increase the retry duration on subsequent retries. There are many strategies for its. See e.g. [exponential back off](https://en.wikipedia.org/wiki/Exponential_backoff) – Jonathan Eunice Jan 10 '16 at 23:20
  • @JohnKeyes the problem with that approach is that you don't know when it will retry and when it won't – Walid Saad Jan 10 '16 at 23:39
  • @WalidSaad +1 I'd go with a backoff between tries, just highlighting some built-in functionality. – John Keyes Jan 10 '16 at 23:42

1 Answers1

2

You could try using requests HTTPAdapter. Something like the following should work:

import time
import httplib

import requests
from requests.adapters import HTTPAdapter


class RetryHTTPAdapter(HTTPAdapter):

    SECONDS_BETWEEN_RETRIES = 5

    def __init__(self, retry_time=120, *args, **kwargs):
        self.retry_time = retry_time
        super(RetryHTTPAdapter, self).__init__(*args, **kwargs)

    def send(self, *args, **kwargs):
        for _ in range(int(self.retry_time / self.SECONDS_BETWEEN_RETRIES)):
            response = super(RetryHTTPAdapter, self).send(*args, **kwargs)
            if response.status_code == httplib.OK:
                break
            time.sleep(self.SECONDS_BETWEEN_RETRIES)
        return response

s = requests.Session()
s.mount('http://', RetryHTTPAdapter(retry_time=180))
s.mount('https://', RetryHTTPAdapter(retry_time=180))

s.get('http://example.com')
TomDotTom
  • 6,238
  • 3
  • 41
  • 39
  • 1
    You could also directly use [HTTPAdapter](https://docs.python-requests.org/en/latest/_modules/requests/adapters/#HTTPAdapter) with urllib's [Retry class](https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.Retry) as `max_retries` parameter. – Hritik Mar 30 '21 at 12:38