In Python: How to remove an object from a list if it is only referenced in that list?

Question

I want to keep track of objects of a certain type that are currently in use. For example: Keep track of all instances of a class or all classes that have been created by a metaclass.

It is easy to keep track of instances like this:

class A():
    instances = []
    def __init__(self):
        self.instances.append(self)

But if an instance is not referenced anywhere outside of that list it will not be needed anymore and I do not want to process that instance in a potentially time consuming loop.

I tried to remove objects that are only referenced in the list using sys.getrefcount.

for i in A.instances:
    if sys.getrefcount(i) <=3: # in the list, in the loop and in getrefcount
        # collect and remove after the loop

The problem I have is that the reference count is very obscure. Opening a new shell and creating a dummy class with no content returns 5 for

sys.getrefcount(DummyClass)

Another idea is to copy the objects then deleting the list and checking which objects have been scheduled for garbage collecting and in the last step removing those objects. Something like:

Copy = copy(A.instances)
del A.instances
A.instances = [i for i in Copy if not copy_of_i_is_in_GC(i)]

The objects don't have to be removed immediately when the reference count goes to 0. I just don't want to waste too much ressources on objects that are not used anymore.

Use a [weak reference](https://docs.python.org/3.5/library/weakref.html) — Barmar, May 14 '16 at 23:47
http://stackoverflow.com/questions/12101958/keep-track-of-instances-in-python — Padraic Cunningham, May 15 '16 at 00:01
http://effbot.org/pyfaq/how-do-i-get-a-list-of-all-instances-of-a-given-class.htm — Padraic Cunningham, May 15 '16 at 00:07

score 7 · Answer 1 · answered May 14 '16 at 23:59

The standard way to solve this problem is through weak references. The basic idea is that you keep a list of weak references to objects instead of the objects themselves, and periodically prune the dead weak references from the list.

For dictionaries and sets, there are some more abstract types such as weakref.WeakKeyDictionary() which can be used when you want to put weak references in more complex places like the keys of a dictionary. These types do not require manual pruning.

score 7 · Accepted Answer · answered May 15 '16 at 00:38

This answer is the same as Kevin's but I was working up an example implementation with weak references and am posting it here. Using weak references solves the problem where an object is referenced by the self.instance list, so it will never be deleted.

One of the things about creating a weak reference for an object is that you can include a callback when the object is deleted. There are issues such as the callback not happening when the program exits... but that may be what you want anyway.

import threading
import weakref

class A(object):
    instances = []
    lock = threading.RLock()

    @classmethod
    def _cleanup_ref(cls, ref):
        print('cleanup') # debug
        with cls.lock:
            try:
                cls.instances.remove(ref)
            except ValueError:
                pass

    def __init__(self):
        with self.lock:
            self.instances.append(weakref.ref(self, self._cleanup_ref))

# test
test = [A() for _ in range(3)]
for i in range(3,-1,-1):
    assert len(A.instances) == i
    if test:
        test.pop()

print("see if 3 are removed at exit")
test = [A() for _ in range(3)]

In the Python shell (3.5.1) the assertion in the test loop fails because the cleanup callback has not been called when the loop reaches the assertion the second time. Printing anything to stdout before the assert statement or after test.pop() fixes it. So, placing "False" before "assert" fixes it, while placing "None" doesn't fix it. — uzumaki, May 16 '16 at 03:16
@uzumaki interesting. I tested with Python 3.4. I'm puzzled why the callback hasn't happened in 3.5. — tdelaney, May 16 '16 at 03:33

score 1 · Answer 3 · answered May 14 '16 at 23:42

1

Try the gc.get_referrers(obj). The gc module Documentation

len(gc.get_referrers(my_obj))

answered May 14 '16 at 23:42

WreckeR

409
3
12

gdlmx · Answer 4 · 2016-05-15T09:47:01.847

Thanks to @Barmar for pointing out to use weakref. We can combine it with the __del__ method to implement a self managing instance list of a class. Thus, the class A in OP's post can be extended as:

from weakref import ref
class A():
    instances = []
    def __init__(self):
        self.instances.append(ref(self))

    @staticmethod
    def __del__():
      if A:
        A.instances = [i for i in A.instances if not i() is None]

Testing

#python2.7
print dict((len(A.instances), A()) for i in range(5)).keys() # 0,1,2,3,4
print len(A.instances) # 0

The destructor __del__ can be declared as a static method or a object-bound method like def __del__(self):, although it is not documented. The latter can stop the object from being destroyed by creating another reference to it. Here I use the static one because there is no need for another reference to the dieing object. The code above is tested in both Python 2.7 and 3.3.

The weakref.ref callback behaves similar to __del__ except that it is bound to the "weakref" object. So if you create multiple weakrefs for the same object with the same callback function, it will be called exact the same time as the number of weakrefs.

Note that [`__del__`](https://docs.python.org/3.5/reference/datamodel.html#object.__del__) shouldn't be a static method. Also `not i() is None` is better written as `i() is not None`. Also, as tdelaney shows `weakref.ref` already provides the means of adding a callback called when the object gets destroyed. — Bakuriu, May 15 '16 at 07:59
@Bakuriu : thanks for comment, pls see the updated answer for discussion about `__del__` and `weakref.ref` — gdlmx, May 15 '16 at 09:50

In Python: How to remove an object from a list if it is only referenced in that list?

4 Answers4