Why set a bound method to python object create a circular reference?

Question

I'm working in Python 2.7 and I fond that issue that puzzling me.

That is the simplest example:

>>> class A(object):
    def __del__(self):
        print("DEL")
    def a(self):
        pass

>>> a = A()
>>> del a
DEL

That is OK like expected... now I'm trying to change the a() method of object a and what happen is that after change it I can't delete a any more:

>>> a = A()
>>> a.a = a.a
>>> del a

Just to do some checks I've print the a.a reference before and after the assignment

>>> a = A()
>>> print a.a
<bound method A.a of <__main__.A object at 0xe86110>>
>>> a.a = a.a
>>> print a.a
<bound method A.a of <__main__.A object at 0xe86110>>

Finally I used objgraph module to try to understand why the object is not released:

>>> b = A()
>>> import objgraph
>>> objgraph.show_backrefs([b], filename='pre-backref-graph.png')

>>> b.a = b.a
>>> objgraph.show_backrefs([b], filename='post-backref-graph.png')

As you can see in the post-backref-graph.png image there is a __self__ references in b that have no sense for me because the self references of instance method should be ignored (as was before the assignment).

Somebody can explain why that behaviour and how can I work around it?

Answer 1

When you write a.a, it effectively runs:

A.a.__get__(a, A)

because you are not accessing a pre-bound method but the class' method that is being bound at runtime.

When you do

a.a = a.a

you effectively "cache" the act of binding the method. As the bound method has a reference to the object (obviously, as it has to pass self to the function) this creates a circular reference.

---

So I'm modelling your problem like:

class A(object):
    def __del__(self):
        print("DEL")
    def a(self):
        pass

def log_all_calls(function):
    def inner(*args, **kwargs):
        print("Calling {}".format(function))

        try:
            return function(*args, **kwargs)
        finally:
            print("Called {}".format(function))

    return inner

a = A()
a.a = log_all_calls(a.a)

a.a()

You can use weak references to bind on demand inside log_all_calls like:

import weakref

class A(object):
    def __del__(self):
        print("DEL")
    def a(self):
        pass

def log_all_calls_weakmethod(method):
    cls = method.im_class
    func = method.im_func
    instance_ref = weakref.ref(method.im_self)
    del method

    def inner(*args, **kwargs):
        instance = instance_ref()

        if instance is None:
            raise ValueError("Cannot call weak decorator with dead instance")

        function = func.__get__(instance, cls)

        print("Calling {}".format(function))

        try:
            return function(*args, **kwargs)
        finally:
            print("Called {}".format(function))

    return inner

a = A()
a.a = log_all_calls_weakmethod(a.a)

a.a()

This is really ugly, so I would rather extract it out to make a weakmethod decorator:

import weakref

def weakmethod(method):
    cls = method.im_class
    func = method.im_func
    instance_ref = weakref.ref(method.im_self)
    del method

    def inner(*args, **kwargs):
        instance = instance_ref()

        if instance is None:
            raise ValueError("Cannot call weak method with dead instance")

        return func.__get__(instance, cls)(*args, **kwargs)

    return inner

class A(object):
    def __del__(self):
        print("DEL")
    def a(self):
        pass

def log_all_calls(function):
    def inner(*args, **kwargs):
        print("Calling {}".format(function))

        try:
            return function(*args, **kwargs)
        finally:
            print("Called {}".format(function))

    return inner

a = A()
a.a = log_all_calls(weakmethod(a.a))

a.a()

Done!

---

FWIW, not only does Python 3.4 not have these issues, it also has WeakMethod pre-built for you.

Answer 2

Veedrac's answer about the bound method keeping a reference to the instance is only part of the answer. CPython's garbage collector knows how to detect and handle cyclic references - except when some object that's part of the cycle has a __del__ method, as mentioned here https://docs.python.org/2/library/gc.html#gc.garbage :

Objects that have __del__() methods and are part of a reference cycle cause the entire reference cycle to be uncollectable, including objects not necessarily in the cycle but reachable only from it. Python doesn’t collect such cycles automatically because, in general, it isn’t possible for Python to guess a safe order in which to run the __del__() methods. (...) It’s generally better to avoid the issue by not creating cycles containing objects with __del__() methods, and garbage can be examined in that case to verify that no such cycles are being created.

IOW : remove your __del__ method and you should be fine.

EDIT: wrt/ your comment :

I use it on the object as function a.a = functor(a.a). When the test is done I would like replace the functor by the original method.

Then the solution is plain and simple:

a = A()
a.a = functor(a.a)
test(a)
del a.a

Until you explicitely bind it, a has no 'a' instance atribute, so it's looked up on the class and a new method instance is returned (cf https://wiki.python.org/moin/FromFunctionToMethod for more on this). This method instance is then called, and (usually) discarded.

Answer 3

As to why Python does this. Technically all objects contain circular references if they have methods. However, garbage collection would take much longer if the garbage collector had to do explicit checks on an objects methods to make sure freeing the object wouldn't cause a problem. As such Python stores the methods separately from an object's __dict__. So when you write a.a = a.a, you are shadowing the method with itself in the a field on the object. And thus, there is an explicit reference to the method which prevents the object from being freed properly.

The solution to your problem is not bother to keep a "cache" of the original method and just delete the shadowed variable when you're done with it. This will unshadow the method and make it available again.

>>> class A(object):
...     def __del__(self):
...         print("del")
...     def method(self):
...         print("method")
>>> a = A()
>>> vars(a)
{}
>>> "method" in dir(a)
True
>>> a.method = a.method
>>> vars(a)
{'method': <bound method A.method of <__main__.A object at 0x0000000001F07940>>}
>>> "method" in dir(a)
True
>>> a.method()
method
>>> del a.method
>>> vars(a)
{}
>>> "method" in dir(a)
True
>>> a.method()
method
>>> del a
del

Here vars shows what's in the __dict__ attribute of an object. Note how __dict__ doesn't contain a reference to itself even though a.__dict__ is valid. dir produces a list of all the attributes reachable from the given object. Here we can see all the attributes and methods on an object and all the methods and attributes of its classes and their bases. This shows that the bound method of a is stored in place separate to where a's attributes are stored.