77

I have been trying to understand how python weak reference lists/dictionaries work. I've read the documentation for it, however I cannot figure out how they work, and what they can be used for. Could anyone give me a basic example of what they do and an explanation of how they work?

(EDIT) Using Thomas's code, when i substitute obj for [1,2,3] it throws:

Traceback (most recent call last):
File "C:/Users/nonya/Desktop/test.py", line 9, in <module>
r = weakref.ref(obj)
TypeError: cannot create weak reference to 'list' object
Nathaniel Ford
  • 20,545
  • 20
  • 91
  • 102
IT Ninja
  • 6,174
  • 10
  • 42
  • 65

4 Answers4

92

Theory

The reference count usually works as such: each time you create a reference to an object, it is increased by one, and whenever you delete a reference, it is decreased by one.

Weak references allow you to create references to an object that will not increase the reference count.

The reference count is used by python's Garbage Collector when it runs: any object whose reference count is 0 will be garbage collected.

You would use weak references for expensive objects, or to avoid circle references (although the garbage collector usually does it on its own).

Usage

Here's a working example demonstrating their usage:

import weakref
import gc

class MyObject(object):
    def my_method(self):
        print 'my_method was called!'

obj = MyObject()
r = weakref.ref(obj)

gc.collect()
assert r() is obj #r() allows you to access the object referenced: it's there.

obj = 1 #Let's change what obj references to
gc.collect()
assert r() is None #There is no object left: it was gc'ed.
Thomas Orozco
  • 53,284
  • 11
  • 113
  • 116
  • 4
    How does it work on list's/dictionary's, this was a great example on classes/functions though. :) – IT Ninja Mar 28 '12 at 13:22
  • 2
    Could you please look at my edit. when substituting a list or dictionary it throws that error. – IT Ninja Mar 28 '12 at 14:04
  • My bad, I totally overlooked this, you can't indeed create weakref to a `list` or another composite type - what is your exact purpose for which you'd need it? – Thomas Orozco Mar 28 '12 at 14:11
  • I am working on an encryption method and i am trying to prevent memory leaks because i am passing values from list to list, and often times these lists are massive but i dont need them for very long. Any thoughts? – IT Ninja Mar 28 '12 at 14:16
  • 5
    Python's `weakref` documentation suggests that you can create a weak reference to a list through subclassling : `class WeakRefableList(list):pass`. I'm however not sure that this will help you in what you want to do, proper scoping should be enough for the garbage collector to handle your problem. You can always use the `gc` mode to track down memory leaks if you want to. – Thomas Orozco Mar 28 '12 at 14:24
  • The "_what they can be used for_" is not answered though. The only _implied_ thing I see in the above **Usage** is to check if I, personally, have (at some point), in my own code, any references to an object, or if I did good programing job and there are not. Usage of this reference will always be ambiguous (is my object there or not?). Is this the only answer to "what they can be used for"? Just to check if my object still exists somewhere in memory? – ilias iliadis Jan 12 '19 at 21:11
  • @ThomasOrozco thanks, but you said this weakref doesn't increase the reference count. Then, how does the program know the 'obj' is there or not there, without the reference count? I can't get the point in the 'Usage' code. – Chan Kim Oct 25 '19 at 08:50
  • Weak references allow you to create references to an object that will not increase the reference count. -- this doesn't make sense. So I'm just not seeing it, really. This says the reference count will always be 0. :) That can't be, so I'm not grasping the wording. What I get is the reference count *can* be increased and that when the deleted references is 0, then GC goes to work. Is that right? – user1585204 Dec 22 '20 at 14:17
54

Just want to point out that weakref.ref does not work for built-in list because there is no __weakref__ in the __slots__ of list. For example, the following code defines a list container that supports weakref.

import weakref

class weaklist(list):
    __slots__ = ('__weakref__',)

l = weaklist()
r = weakref.ref(l)
Ricardo Branco
  • 5,740
  • 1
  • 21
  • 31
jichi
  • 6,333
  • 1
  • 31
  • 25
16

The point is that they allow references to be retained to objects without preventing them from being garbage collected.

The two main reasons why you would want this are where you do your own periodic resource management, e.g. closing files, but because the time between such passes may be long, the garbage collector may do it for you; or where you create an object, and it may be relatively expensive to track down where it is in the programme, but you still want to deal with instances that actually exist.

The second case is probably the more common - it is appropriate when you are holding e.g. a list of objects to notify, and you don't want the notification system to prevent garbage collection.

Marcin
  • 48,559
  • 18
  • 128
  • 201
  • 1
    Perhaps 3 reasons? third the breaking out of the cyclic references. – dashesy Sep 15 '15 at 01:08
  • 2
    @dashesy has it right. The most common (_arguably, the only safe_) use case for weak references is to explicitly break reference cycles. Although the Python garbage collector can and eventually will break such cycles anyway, it's substantially less space- and time-intensive to prevent such cycles from forming in the first place. All other uses of weak references are fragile at best and should _absolutely_ be treated with a modicum of fear and respect. – Cecil Curry Mar 11 '16 at 02:39
4

Here is the example comparing dict and WeakValueDictionary:

class C: pass
ci=C()
print(ci)

wvd = weakref.WeakValueDictionary({'key' : ci})
print(dict(wvd), len(wvd)) #1
del ci
print(dict(wvd), len(wvd)) #0

ci2=C()
d=dict()
d['key']=ci2
print(d, len(d))
del ci2
print(d, len(d))

And here is the output:

<__main__.C object at 0x00000213775A1E10>
{'key': <__main__.C object at 0x00000213775A1E10>} 1
{} 0
{'key': <__main__.C object at 0x0000021306B0E588>} 1
{'key': <__main__.C object at 0x0000021306B0E588>} 1

Note how in the first case once we del ci the actual object will be also removed from the dictionary wvd.

In the case or regular Python dictionary dict class, we may try to remove the object but it will still be there as shown.

Note: if we use del, we do not to call gc.collect() after that, since just del effectively removes the object.

prosti
  • 42,291
  • 14
  • 186
  • 151