Try/Except Every Method in Class?

Question

I'm currently working with Google's BigQuery API, which when called, occasionally gives me:

apiclient.errors.HttpError: <HttpError 500 when requesting https://www.googleapis.com/bigquery/v2/projects/some_job?alt=json returned "Unexpected. Please try again.">

It's kind of a silly thing to return, but anyway, when I get this for any method called, I'd want to just sleep a second or two and then try again. Basically, I'd want to wrap every method with something like:

def new_method
    try:
        method()
    except apiclient.errors.HttpError, e:
        if e.resp.status == 500:
            sleep(2)
            new_method()
        else:
            raise e

What's a good way of doing this?

I don't want to explicitly redefine every method in the class. I just want to apply something automatically to every method in the class, so I'm covered for the future. Ideally, I'd take a class object, o, and make a wrapper around it that redefines every method in the class with this try except wrapper so I get some new object, p, that automatically retries when it gets a 500 error.

Possible duplicate of [How can I reuse exception handling code for multiple functions in Python?](https://stackoverflow.com/questions/28965795/how-can-i-reuse-exception-handling-code-for-multiple-functions-in-python) — Stevoisiak, May 23 '18 at 16:00

score 10 · Accepted Answer · edited Jun 03 '14 at 21:50

10

Decorators are perfect for this. You can decorate each relevant method with a decorator like this one:

(Note using recursion for retries is probably not a great idea ...)

def Http500Resistant(func):
    num_retries = 5
    @functools.wraps(func)
    def wrapper(*a, **kw):
        sleep_interval = 2
        for i in range(num_retries):
            try:
                return func(*a, **kw)
            except apiclient.errors.HttpError, e:
                if e.resp.status == 500 and i < num_retries-1:
                    sleep(sleep_interval)
                    sleep_interval = min(2*sleep_interval, 60)
                else:
                    raise e    
    return wrapper

class A(object):

    @Http500Resistant
    def f1(self): ...

    @Http500Resistant
    def f2(self): ...

To apply the decorator to all methods automatically, you can use yet-another-decorator, this time, decorating the class:

import inspect
def decorate_all_methods(decorator):
    def apply_decorator(cls):
        for k, f in cls.__dict__.items():
            if inspect.isfunction(f):
                setattr(cls, k, decorator(f))
        return cls
    return apply_decorator

and apply like this:

@decorate_all_methods(Http500Resistant)
class A(object):
    ...

Or like:

class A(object): ...
A = decorate_all_methods(Http500Resistant)(A)

edited Jun 03 '14 at 21:50

Robert Harvey

178,213
47
333
501

answered Jun 03 '14 at 21:24

shx2

61,779
13
130
153

1

A trick I use here is to increase the interval between retries after each failure, to reduce hammering on an already-overloaded server. For example, initialize retry_interval to 1.0 at the top of the loop, multiply it by 2.0 after each failure, and sleep for that value. – Russell Borogove Jun 03 '14 at 21:29
I don't want to explicitly redefine every method in the class. I just want to apply something automatically to every method in the class, so I'm covered for the future. Ideally, I'd take a class object, o, and make a wrapper around it that redefines every method in the class with this try except wrapper so I get some new object, p, that automatically retries when it gets a 500 error. – Eli Jun 03 '14 at 21:32
@RussellBorogove, I'd do that if the error was specifically about getting throttled, which Google provides as well, but since this is just an unknown error, I'd rather sleep a static value and retry a few times before failing. – Eli Jun 03 '14 at 21:36
Is there a way to apply this to an instance of a class? I'm running a function that just returns to me `return discovery.build('bigquery', 'v2', http=http)`, and I just want to decorate the instance returned in the fashion you mentioned. – Eli Jun 03 '14 at 22:14
@Eli a quick-and-dirty way would be: `obj.__class__ = Http500Resistant(obj.__class__)` (would only work in some cases). But if it makes your stomach rumble, I'm sure you can write something which decorates all methods of an object. It wouldn't be that much different than the `decorate_all_methods` decorator in my answer. – shx2 Jun 03 '14 at 22:20

score 3 · Answer 2 · answered Jun 03 '14 at 21:35

As other answers pointed out, you can accomplish this with decorators:

def retry(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        # retry logic
    return wrapper

If you want to automatically apply this decorator to all methods of a class, you can use a metaclass for this:

class Meta(type):
    def __new__(cls, name, bases, attrs):
        for n in attrs:
            if inspect.isfunction(attrs[n]):
                attrs[n] = retry(attrs[n])
        return super(Meta, cls).__new__(cls, name, bases, attrs)

class Api(object):
    __metaclass__ = Meta

    def function_with_retry_applied(self):
        raise HttpError(500)

score 1 · Answer 3 · answered Jun 03 '14 at 22:39

The ideas here are from @shx2's answer, but since what I really wanted was a way to apply something to every function in an object and not a class, I'm supplying this for anyone with the same question in the future:

def bq_methods_retry(func):
    num_retries = 5
    @functools.wraps(func)
    def wrapper(*a, **kw):
        sleep_interval = 2
        for i in xrange(num_retries):
            try:
                return func(*a, **kw)
            except apiclient.errors.HttpError, e:
                if e.resp.status == 500 and i < num_retries-1:
                    time.sleep(sleep_interval)
                    sleep_interval = min(2*sleep_interval, 60)
                else:
                    raise e
    return wrapper


def decorate_all_bq_methods(instance, decorator):
    for k, f in instance.__dict__.items():
        if inspect.ismethod(f):
            setattr(instance, k, decorator(f))
    return instance

Now, when you create a new BQ service, just apply decorate_all_bq_methods() to it as:

service = discovery.build('bigquery', 'v2', http=http)
#make all the methods in the service retry when appropriate
retrying_service = decorate_all_bq_methods(service, bq_methods_retry)

Try/Except Every Method in Class?

3 Answers3

Linked