python flow control for conditionally retrying an operation

Question

I'm attempting an operation in Python, and if it fails in certain ways I'd like to retry up to 10 times. If it fails in any other way I'd like it to fail immediately. After 10 retries I'd like all failures to propagate to the caller.

I've been unable to code the flow control in a satisfying way. Here's one example of the behavior (but not the style!) I want:

 def run():
    max_retries = 10
    for retry_index in range(max_retries):
        try:
            result = run_operation()
        except OneTypeOfError:
            if retry_index < max_retries - 1:
                continue
            else:
                raise

        if result.another_type_of_error:
            if retry_index < max_retries - 1:
                continue
            else:
                raise AnotherTypeOfError()

        try:
            result.do_a_followup_operation()
        except AThirdTypeOfError:
            if retry_index < max_retries - 1:
                continue
            else:
                raise

        return result

    raise Exception("We don't expect to end up here")

At first I thought I could just refactor this so the retry logic is separate from the error handling logic. The problem is that if, for example, OneTypeOfError is raised by result.do_a_followup_operation(), I don't want to retry at all in that case. I only want to retry in the specific circumstances coded above.

So I thought perhaps I could refactor this into a function which returns the result, a raised Exception (if any), and a flag indicating whether it's a retryable exception. Somehow that felt less elegant than the above to me.

I'm wondering if there are any flow control patterns in Python which might help here.

You can [have multiple except blocks](https://stackoverflow.com/questions/6095717/python-one-try-multiple-except) — , Sep 29 '17 at 16:09

kchawla-pi · Answer 1 · 2017-09-29T16:50:09.277

Edit: Ah I didn't realize that your code indicated you didn't use multiple except blocks, as pointed out by JETM

Here's your quick primer on ExceptionHandling:

try:
    # do something
except OneTypeOfError as e:
    # handle one type of error one way
    print(e)  # if you want to see the Exception raised but not  raise it
except AnotherTypeOfError as e:
    # handle another type of error another way
    raise e('your own message')
except (ThirdTypeOfError, FourthTypeOfError) as e:
    # handle error types 3 & 4 the same way
    print(e)  # if you want to see the Exception raised but not  raise it
except:  # DONT DO THIS!!!
    '''
    Catches all and any exceptions raised.
    DONT DO THIS. Makes it hard to figure out what goes wrong.
    ''' 
else:
    # if the try block succeeds and no error is raisedm then do this.
finally:
    '''
    Whether the try block succeeds or fails and one of the except blocks is activated.
    Once all those are done with, finally run this block.
    This works even if your program crashed and so is great for cleaning up for example.
    '''

I did this once a long time ago, and recursively called the same function in one kind of exception but not another. I also passed the retry index and max_retries variable to the function, which meant adding those as parameters.

The other way would be to place the entire thing in a for loop of max_retries and add a break in all except blocks for the exceptions where you don't want a retry.

Finally, instead of a for loop, you can put the entire thing in a while loop, insert an increment condition in except block for one type of exception and make the while condition false in except block for other exceptions.

score 0 · Accepted Answer · answered Sep 29 '17 at 16:13

You can use a specific exception class and recursion to be a little more dry. Sth along these lines:

class Retry(Exception):
    def __init__(self, e):
        super(Exception, self).__init__()
        self.e = e

def run(max_retries=10, exc=None):
    if max_retries <= 0:
        raise exc or Exception('Whatever')
    try:
        try:
            result = run_operation()
        except OneTypeOfError as e:
            raise Retry(e)
        if result.another_type_of_error:
            raise Retry(AnotherTypeOfError())
        try:
            result.do_a_followup_operation()
        except AThirdTypeOfError as e:
            raise Retry(e)
    except Retry as r:
        return run(max_retries=max_retries-1, exc=r.e)
    else:
        return result

An iterative solution with a given number of iterations seems semantically questionable. After all, you want the whole thing to succeed and a retry is a fallback. And having run out of retries sounds like a base case to me.

Aha this does indeed feel much more elegant to me - thank you! — nonagon, Sep 29 '17 at 20:03

mkrieger1 · Answer 3 · 2017-09-29T16:38:45.680

If I understand this correctly, you could do it as follows by changing two things:

Counting tries_left down from 10 instead of retry_index up from 0 reads more naturally and lets you exploit that positive numbers are truthy.
If you changed (or wrapped) run_operation() such that it would already raise AnotherTypeOfError if result.another_error is true, you could combine the first two except blocks.

The code can optionally be made slightly more dense by omitting the else after raise (or after continue if you choose to test for if tries_left instead) – the control flow is diverted at that point anyway –, and by putting a simple statement on the same line as a bare if without else.

for tries_left in range(10, -1, -1):
    try:
        result = run_operation()
    except OneTypeOfError, AnotherTypeOfError:
        if not tries_left: raise
        continue

    try:
        result.do_a_followup_operation()
    except AThirdTypeOfError:
        if not tries_left: raise
        continue

    return result

python flow control for conditionally retrying an operation

3 Answers3