Pythonic way of retry running a function

Question

How a Python Professinal would retry running a function that will request a web service(the webservice sometimes fails)

The function:

def some_request(uri):
    try:
        req = requests.post('http://someuri.com', {'test': 'yes'})
    except Exception as e:
        return False
    return {'id': req.json()['value'], 'other': req.json()['other']}

You handle the retry with a while or other python idiom?

Give me a clue on how to do it the right way.

Increase the retry count for the `requests` library and have it handle the problem. — Martijn Pieters, Feb 14 '14 at 18:03
And rarely do you need to catch a blanket `Exception`; `requests.exceptions` has all the specific exceptions that can be thrown, pick one from there instead. — Martijn Pieters, Feb 14 '14 at 18:04

Omid Raha · Accepted Answer · 2014-02-14T20:30:57.203

Define retry utility:

# Retry utility   
# set logging for `retry` channel
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger('retry')

# Define Exception class for retry
class RetryException(Exception):
    u_str = "Exception ({}) raised after {} tries."

    def __init__(self, exp, max_retry):
        self.exp = exp
        self.max_retry = max_retry    
    def __unicode__(self):
        return self.u_str.format(self.exp, self.max_retry)
    def __str__(self):
        return self.__unicode__()

# Define retry util function
def retry_func(func, max_retry=10):
    """
    @param func: The function that needs to be retry
    @param max_retry: Maximum retry of `func` function, default is `10`
    @return: func
    @raise: RetryException if retries exceeded than max_retry
    """
    for retry in range(1, max_retry + 1):
        try:
            return func()
        except Exception, e:
            logger.info('Failed to call {}, in retry({}/{})'.format(func.func,
                                                           retry, max_retry))
    else:
        raise RetryException(e, max_retry)

Use it:

from functools import partial
import requests

def some_request(uri, post):
    req = requests.post(uri, post)
    return req

uri = 'http://someuri.com'
post = {'test': 'yes'}

try:
    retry_func(partial(some_request, uri, post), max_retry=3)
except RetryException, e:
    print(e)

Output:

INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): someuri.com
INFO:retry:Failed to call <function some_request at 0x7faaba2318c0>, in retry(1/3)
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): someuri.com
INFO:retry:Failed to call <function some_request at 0x7faaba2318c0>, in retry(2/3)
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): someuri.com
INFO:retry:Failed to call <function some_request at 0x7faaba2318c0>, in retry(3/3)
Exception (HTTPConnectionPool(host='someuri.com', port=80): Max retries exceeded with url: / (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)) raised after 3 tries.

May be overkill, but this is most definitely the best and most complete answer. Consider commenting your code for readability and this should be an obvious checkmark — Adam Smith, Feb 14 '14 at 19:54

score 6 · Answer 2 · answered Feb 14 '14 at 19:59

this definitely calls for a decorator.

you'd probably want something like:

import functools
def retry(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        retExc = None
        for i in xrange(4):
            try:
                return func(*args, **kwargs)
            except Exception, e:
                retExc = e
        raise retExc
    return wrapper

and then to run your code, just use:

@retry
def some_request(...):
    ...

you could make it fancier by adding in a little sleep, adding a numTimes to retry, etc, but this will do it.

score 6 · Answer 3 · edited Jul 18 '23 at 09:58

Here's a generalized decorator that lets you specify the number of retries and also the exception type to ignore:

from functools import wraps
def retry(count=5, exc_type=Exception):
    def decorator(func):
        @wraps(func):
        def result(*args, **kwargs):
            for _ in range(count):
                try:
                    return func(*args, **kwargs)
                except exc_type:
                    pass
                raise
         return result
     return decorator

Use it like this:

@retry(count=10, exc_type=IOError)
def read_file():
    pass # do something

score 5 · Answer 4 · answered Feb 14 '14 at 19:48

5

The pattern I use retries a limited number of times, sleeping for a short, exponentially increasing time interval between each attempt, and finally raising the last-seen exception after repeated failure:

def wrappedRequest( self, uri, args ):
    lastException = None 
    interval = 1.0
    for _ in range(3):
        try:
            result = requests.post(uri, args)
            return result
        except Exception as e:
            lastException = e

        time.sleep(interval)
        interval *= 2.0
    raise lastException

(I actually specifically check for HTTPError, URLError, IOError, and BadStatusLine exceptions rather than the broad net of Exception.)

answered Feb 14 '14 at 19:48

Russell Borogove

18,516
4
43
50

I like your answer, but your code has problems: 1) if it succeeds on second attempt, it will raise an exception from the first failed attempt; 2) if it succeeds from the first attempt, it will `raise None` that is an error in Python; 3) it will uselessly sleep on the last attempt – wobmene Oct 24 '14 at 14:47
1

Not so. If it succeeds on any attempt, it returns from inside the try block, skipping both the `raise` and the `sleep`. – Russell Borogove Oct 24 '14 at 18:09

Pythonic way of retry running a function

4 Answers4

Linked