Using closure to decorate a class's methods

Question

I need to generate a class that would mimic another class's method set and behave like the latter via a proxy. e.g. If Base is the class to mimic and Deleguate is the class which needs to act as Base then:

b = Base(args)
b.any_function()

is strictly equivalent to

d = Deleguate(b)
d.any_function()

If Deleguate uses a function that already exists in Base, it won't be overwritten. It's the kind of behavior you expect with inheritance and method overriding. Inheritance is not an option in the context of the project I'm working on (among other constraint, I don't have access to the factory code). And that's what makes things complicated.

I therefore decided to code a "proxy" decorator :

import inspect

def proxy(bridge, target):
    def proxyfy(cls):
        for _, func in inspect.getmembers(target, predicate=inspect.ismethod):
            fname = func.__name__
            if fname in cls.__dict__:
                print 'ignoring %s.%s' % (cls, fname)
                continue
            print 'adding %s.%s' % (cls, fname)
            def proxy_func(self, *args, **kwargs):
                print 'calling %s.%s.%s' % (cls, bridge, fname)
                bridge_member = getattr(self, bridge)
                return getattr(bridge_member, fname)(*args, **kwargs)
            setattr(cls, fname, proxy_func)
        return cls
    return proxyfy

class Base(object):
    def __init__(self, i):
        self._i = i

    def __bar(self):
        print 0

    def foo(self):
        print self._i

    def foo2(self):
        print 2 * self._i


@proxy('_proxy', Base)
class Deleguate(object):
    def __init__(self, base):
        self._proxy = base

    def foo2(self):
        print 4 * self._proxy._i

d = Deleguate(Base(1))
d.__bar() # d._proxy.__bar()
d.foo()   # d._proxy.foo()
d.foo2()  # d.foo2()

I get the following output :

adding <class '__main__.Deleguate'>.__bar
ignoring <class '__main__.Deleguate'>.__init__
adding <class '__main__.Deleguate'>.foo
ignoring <class '__main__.Deleguate'>.foo2
calling <class '__main__.Deleguate'>._proxy.foo2
2
calling <class '__main__.Deleguate'>._proxy.foo2
2
4

I thought that setattr(cls, fname, proxy_func) would assign a new closure, but the arguments are overwritten at each loop step and only the arguments of the last function foo2 are kept. Therefore calling any "generated" function of Deleguate uses foo2 arguments...

Why are the closure arguments being overwritten ? Is there a way to generate that kind of proxy code ? The expected output is :

adding <class '__main__.Deleguate'>.__bar
ignoring <class '__main__.Deleguate'>.__init__
adding <class '__main__.Deleguate'>.foo
ignoring <class '__main__.Deleguate'>.foo2
calling <class '__main__.Deleguate'>._proxy.__bar
0
calling <class '__main__.Deleguate'>._proxy.foo
1
4

`fname` in `proxy_func()` is a *closure*; it is not looked up until you call `proxy_func()`, at which point it's value is still bound to the last value (`foo2` in this case), **not** the value it was bound to when you created the nested function. — Martijn Pieters, Oct 26 '13 at 14:07
The workaround is to create a local variable somewhere to bind the value to in the loop; a separate factory function would do that, or by giving `proxy_func()` a keyword argument that binds the `fname` as a default to the function object. — Martijn Pieters, Oct 26 '13 at 14:10
@MartijnPieters I did that. But it fails with `AttributeError: 'Base' object has no attribute '__bar'` on `d.__bar() # d._proxy.__bar()`. :'( — thefourtheye, Oct 26 '13 at 14:13
@MartijnPieters Its been half an hour since I started working on this. :( I changed the signature like this `def proxy_func(self, fname=fname, *args, **kwargs):`. — thefourtheye, Oct 26 '13 at 14:14
@thefourtheye: That is because double-underscore names are mangled; see http://docs.python.org/2/reference/expressions.html#atom-identifiers — Martijn Pieters, Oct 26 '13 at 14:17
@MartijnPieters Damn. Without you, I would never have figured this one out. Thanks a lot :) Please add an answer, it will really be useful for future reference :) — thefourtheye, Oct 26 '13 at 14:19
@thefourtheye: Nah, this is a dupe, as it stands. I already linked to my canonical answer on the subject. — Martijn Pieters, Oct 26 '13 at 14:20
@MartijnPieters Agreed. But, even if he figures out the closure thing, it would be difficult to understand the mangling part, I believe. — thefourtheye, Oct 26 '13 at 14:22
@thefourtheye: But that's a side-issue; a comment suffices to cover that issue. — Martijn Pieters, Oct 26 '13 at 14:39
@MartijnPieters Thanks for the reference. I wasn't looking for the right keywords — user2221662, Oct 26 '13 at 17:32

unutbu · Accepted Answer · 2013-10-26T14:32:42.507

Functions create closures, loops do not. The variable name fname is a local variable in proxyfy. The nested function proxy_func refers to this local variable. But at the time when the nested function is called, the for-loop

    for _, func in inspect.getmembers(target, predicate=inspect.ismethod):

has completed, and the local variable fname references its last value at the end of the loop, which happens to be 'foo2'. So no matter what method you call, each proxy_func ends up calling foo2.

To bind different values of fname to each proxy_func, you could use a new keyword parameter, bname with a default value. The default value is bound to the function at definition-time not at the time the function in run. So if use

    for bname, func in inspect.getmembers(target, predicate=inspect.ismethod):

and use this bname as the default value:

        def proxy_func(self, bname=bname, *args, **kwargs):

then each proxy_func will call the appropriate bname.

So, with minimal changes to your code, you could add a keyword parameter with default value to proxy_func to remember the current method name:

def proxy(bridge, target):
    def proxyfy(cls):
        for bname, func in inspect.getmembers(target, predicate=inspect.ismethod):
            fname = func.__name__
            if fname in cls.__dict__:
                print 'ignoring %s.%s' % (cls, fname)
                continue
            print 'adding %s.%s' % (cls, fname)
            def proxy_func(self, bname=bname, *args, **kwargs):
                print 'calling %s.%s.%s' % (cls, bridge, bname)
                bridge_member = getattr(self, bridge)
                return getattr(bridge_member, bname)(*args, **kwargs)
            setattr(cls, fname, proxy_func)
        return cls
    return proxyfy

However, I think using __getattr__ might be easier:

def proxy(bridge):
    def proxyfy(cls):
        def __getattr__(self, attr):
            target = getattr(self, bridge)
            if attr.startswith('__') and not attr.endswith('__'):
                # unmangle
                attr = '_{}{}'.format(type(target).__name__, attr)
            return getattr(target, attr)
        setattr(cls, '__getattr__', __getattr__)
        return cls
    return proxyfy

Here's a runnable example:

import inspect

def proxy(bridge, target):
    def proxyfy(cls):
        for bname, func in inspect.getmembers(target, predicate=inspect.ismethod):
            fname = func.__name__
            if fname in cls.__dict__:
                print 'ignoring %s.%s' % (cls, fname)
                continue
            print 'adding %s.%s' % (cls, fname)
            def proxy_func(self, bname=bname, *args, **kwargs):
                print 'calling %s.%s.%s' % (cls, bridge, bname)
                bridge_member = getattr(self, bridge)
                return getattr(bridge_member, bname)(*args, **kwargs)
            setattr(cls, fname, proxy_func)
        return cls
    return proxyfy

def proxy(bridge):
    def proxyfy(cls):
        def __getattr__(self, attr):
            target = getattr(self, bridge)
            if attr.startswith('__') and not attr.endswith('__'):
                # unmangle
                attr = '_{}{}'.format(type(target).__name__, attr)
            return getattr(target, attr)
        setattr(cls, '__getattr__', __getattr__)
        return cls
    return proxyfy

class Base(object):
    def __init__(self, i):
        self._i = i

    def __bar(self):
        print 0

    def foo(self):
        print self._i

    def foo2(self):
        print 2 * self._i


# @proxy('_proxy', Base)
@proxy('_proxy')
class Delegate(object):
    def __init__(self, base):
        self._proxy = base

    def foo2(self):
        print 4 * self._proxy._i

d = Delegate(Base(1))
d.__bar() # d._proxy.__bar()
d.foo()   # d._proxy.foo()
d.foo2()  # d.foo2()

Thank you! Your very first sentence sum it all up. Plus you made the distinction between `bname` and `fname`, which matters when playing with mangled methods. — user2221662, Oct 26 '13 at 17:27

Using closure to decorate a class's methods

1 Answers1