How do I use a non-exponential backoff strategy for retries in python requests?

Question

I want to make use of the requests module backoff strategy outlined at https://stackoverflow.com/a/35504626/1021819, the code reproduced from that answer being:

import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

s = requests.Session()

retries = Retry(total=5,
                backoff_factor=0.1,
                status_forcelist=[ 500, 502, 503, 504 ])

s.mount('http://', HTTPAdapter(max_retries=retries))

s.get('http://httpstat.us/500')

[This link gives further useful info: https://www.peterbe.com/plog/best-practice-with-retries-with-requests]

According to the documentation for backoff_factor at https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#module-urllib3.util.retry,

A backoff factor to apply between attempts after the second try (most errors are resolved immediately by a second try without a delay). urllib3 will sleep for: {backoff factor} * (2 ** ({number of total retries} - 1)) seconds. If the backoff_factor is 0.1, then sleep() will sleep for [0.0s, 0.2s, 0.4s, …] between retries. It will never be longer than Retry.BACKOFF_MAX.

How can I elegantly implement a non-exponential backoff e.g. linear?

Thanks as ever!

score 1 · Accepted Answer · answered Mar 09 '20 at 12:04

Although I was deep in enemy territory, I ended up superclassing Retry and overloading Retry.get_backoff_time(). The new class MyRetry now takes an optional lambda specifying how to calculate the backoff time.

import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter
from itertools import takewhile


def linear_backoff(backoff_factor, consecutive_errors_len):
    return backoff_factor * float(consecutive_errors_len - 1)

class MyRetry(Retry):

    def __init__(self, *args, **kwargs):
        self.backoff_func = kwargs.pop('backoff_func', None)
        super(MyRetry, self).__init__(*args, **kwargs)

    def get_backoff_time(self):
        """ Formula for computing the current backoff
        :rtype: float
        """
        # We want to consider only the last consecutive errors sequence (Ignore redirects).
        consecutive_errors_len = len(
            list(
                takewhile(lambda x: x.redirect_location is None, reversed(self.history))
            )
        )
        if consecutive_errors_len <= 1:
            return 0

        if self.backoff_func is not None:
            backoff_value = self.backoff_func(self.backoff_factor, consecutive_errors_len)
        else:
            backoff_value = self.backoff_factor * (2 ** (consecutive_errors_len - 1))
        return min(self.BACKOFF_MAX, backoff_value)

retries = MyRetry(total=5,
                 backoff_factor=0.1,
                 status_forcelist=[500, 502, 503, 504],backoff_func=lambda x,y:linear_backoff(x,y))

s = requests.Session()
s.mount('http://', HTTPAdapter(max_retries=retries))
s.get('http://httpstat.us/500')

Hey Presto!

The lambda is probably not necessary. It would be nice if the original Retry class could be made to take a lambda anyway.

How do I use a non-exponential backoff strategy for retries in python requests?

1 Answers1