Using the observer pattern to delete an Observer when a Subject has been "deleted" in Python

Question

I am attempting to implement a very simple observer pattern in python.

Here is my Observer class (it's really just an interface and I guess I don't actually need it):

class Observer():
    def update(self,subject,message): pass

And my Subject class (aka Observable, but I prefer Subject):

class Subject():
    def __init__(self):
        self.observers = []
    def registerObserver(self, observer):
        if observer not in self.observers:
            self.observers.append(observer)
    def removeObserver(self, observer):
        self.observers.remove(observer)
    def notifyObservers(self, message = None):
        for observer in self.observers:
            observer.update(self,message)

Class A contains a nested DelNotifier class, which is a subclass of Subject. The idea that when a class A object is deleted (actually garbage collected, since it is in the __del__ method), A.DelNotifier will notify all of its observers of the deletion.

class A():
    def __init__(self, name):
        self.name = name
        self.delNotifier = A.DelNotifier(self)
    class DelNotifier(Subject):
        def __init__(self, outer):
            super(A.DelNotifier,self).__init__()
            self.outer = outer
        def notifyObservers(self):
            Subject.notifyObservers(self,"This is Class A object " + self.outer.name + ": I'm dying!")
    def registerB(self,observer):
        if not isinstance(observer,B): raise ValueError("Can only register Class B objects with Class A.")
        self.delNotifier.registerObserver(observer.Aobserver)
    def deleteme(self):
        print("Must notify observers of my impending doom first...")
        self.delNotifier.notifyObservers()
    def __str__(self):
        return "Class A object " + self.name
    def __del__(self):
        self.deleteme()
        print("Done notifying everyone, time to go gentle into that good night.")

Class B contains a nested AObserver class, which is a subclass of Observer and will receive the message from the class A.DelNotifier subject when an A has been deleted (again, in actuality this happens when the A object has been garbage collected):

class B():
    def __init__(self, name, a):
        self.name = name
        self.Aobserver = B.AObserver(self)
        a.registerB(self)
    class AObserver(Observer):
        def __init__(self,outer):
            super(B.AObserver,self).__init__()
            self.outer = outer
        def update(self,subject,message):
            print(str(self.outer) + " received message: '" + str(message) + "'")
            print("Time for", self.outer, "to die, too.")
            self.outer.__del__()
    def __str__(self):
        return "Class B object " + self.name
    def __del__(self):
        print("This is " + str(self) + ": now I'm dying, too!")

This design works when I call __del__() directly, however, some of the objects seem to be gc'd a second time when the session exits:

>>> a = A('a')
>>> b1 = B('b1', a)
>>> b2 = B('b2', a)
>>> a.__del__()
Must notify observers of my impending doom first...
Class B object b1 received message: 'This is Class A object a: I'm dying!'
Time for Class B object b1 to die, too.
This is Class B object b1: now I'm dying, too!
Class B object b2 received message: 'This is Class A object a: I'm dying!'
Time for Class B object b2 to die, too.
This is Class B object b2: now I'm dying, too!
Done notifying everyone, time to go gentle into that good night.
>>> exit()
Must notify observers of my impending doom first...
Class B object b1 received message: 'This is Class A object a: I'm dying!'
Time for Class B object b1 to die, too.
This is Class B object b1: now I'm dying, too!
Class B object b2 received message: 'This is Class A object a: I'm dying!'
Time for Class B object b2 to die, too.
This is Class B object b2: now I'm dying, too!
Done notifying everyone, time to go gentle into that good night.
This is Class B object b1: now I'm dying, too!
This is Class B object b2: now I'm dying, too!

Another problem, and I think this is more important, is that when I del a class A item from a list, the item is not immediately garbage collected and I cannot be sure that any registered B items have been deleted:

>>> b1 = B('b1',a[0])
>>> b2 = B('b2',a[0])
>>> del a[0]
## Note that items are not deleted until session exits
>>> exit()
Must notify observers of my impending doom first...
Class B object b1 received message: 'This is Class A object a: I'm dying!'
Time for Class B object b1 to die, too.
This is Class B object b1: now I'm dying, too!
Class B object b2 received message: 'This is Class A object a: I'm dying!'
Time for Class B object b2 to die, too.
This is Class B object b2: now I'm dying, too!
Done notifying everyone, time to go gentle into that good night.
##Note that the class B objects get gc'd a second time....???
This is Class B object b1: now I'm dying, too!
This is Class B object b2: now I'm dying, too!

In addition to these problems, I am aware of the many many problems inherent in relying on the __del__ method to do anything other than cleaning up an object after it has been gc'd, and that it should probably be avoided for the purposes I am trying to employ. But I don't know of another way.

What would be a better way to do this? I have thought about trying to use a context manager (with) to delete things after I'm done using them, but I have no experience doing that. If that would be a good option, how would I go about doing that? What would it look like?

EDIT: Clarification of desired behavior

I'll attempt to clear up some of the (understandable) confusion.

I've simplified the code quite a bit above, but B is an object that depends on an object A. If a B's A goes away, that B should go away. I'll have some container (using a list here) of As and of Bs:

As = [A('a'+str(i)) for i in range(10)]
Bs = [B('b'+str(i),As[i]) for i in range(10)] #Bs made of As
del As[0] #whoops, don't need As[0] anymore
assert Bs[0] is None #ERROR!
#or using pop:
As.pop(0)
assert Bs[0] is None #ERROR!

Also see my previous question from the other day which helped lead me to this whole idea of using the observer pattern in the first place.

Answer 1

Caveat: I'm working in 2.7.3, so some of this may be a little different.

I've got some code that seems to work a bit better than what you have. Hopefully it'll raise some relevant points.

First, the observer interface shows a deleting event that will be called when a subject is going away. As you've noted, this isn't really necessary but I like to declare my interfaces explicitly.

class Observer(object):
    def update(self,subject,message): pass

    def deleting(self,subject):
        ''' the subject is being deleted '''
        pass

Next, the Subject class will be responsible for handling the __del__, and notifying all observers. This is a bit simplified, and takes the code away from A, and gets rid of all internal classes from your initial example. In the Subject's __del__ method, I copy the observers, clear them out (to remove all references), then call the deleting method to allow the observers to delete themselves.

class Subject(object):
    def __init__(self):
        self.observers = []

    def __del__(self, ):
        ''' on deletion, notify the observers '''
        print "Subject.__del__".format(self)

        # copy the observer list, then remove all references to the observers
        obs = self.observers[:]
        self.observers = []

        # notify all observers that we were deleted
        for o in obs:
            o.deleting(self)

    def registerObserver(self, observer):
        if observer not in self.observers:
            self.observers.append(observer)

I've left out the removeObserver and notifyObservers because I'm not using them in this solution (clearly they are useful otherwise).

The class A is then just a simple Subject and doesn't provide anything but a naming functionality here. I've removed the restriction that only Bs can register to As for simplicity, and I don't think it's necessary to solve the problem.

class A(Subject):
    ''' A is just a subject '''
    def __init__(self, name):
        super(A,self).__init__()
        self.name = name

    def __str__(self):
        return "A[name={0}]".format(self.name)

Then B, as an observer automatically adds itself as an observer of A, on creation. And when the observed subject is deleted, notification is received through the deleting method. Since there is only one observed subject, I assume it is the A passed into __init__ and delete self. It's important (at least in 2.7) to not store references to a within B, or it appears that __del__ may never get called by the gc (this could be my misunderstanding of the gc).

class B(Observer):
    ''' class B is an observer of A '''

    def __init__(self, name, a):
        self.name = name

        # don't keep a reference to 'a', or 'a' will not be deleted!
        a.registerObserver(self)

    def __str__(self):
        return "B[name={0}]".format(self.name)

    def __del__(self):
        print("{0}.__del__".format(self))

    def deleting(self, subject):
        ''' notification from the subject (A) that it is being deleted. I'm
        assuming here that the subject is actually the one we registered in
        __init__, since I couldn't store a reference or else __del__ would
        not have been called! '''
        print "B.deleting, subject={0}".format(subject)

        del self

With this code, I get the output:

>>> from main import A, B
>>> a = A('a')
>>> B('b1', a)
<main.B object at 0x00000000022938D0>
>>> B('b2', a)
<main.B object at 0x0000000002293B70>

>>> del a
Subject.__del__
B.deleting, subject=A[name=a]
B.deleting, subject=A[name=a]
B[name=b1].__del__

>>> import gc
>>> gc.collect()
B[name=b2].__del__
0

which seems more like what you want, except that b2 isn't immediately garbage collected.

EDIT: Added example based on further clarification of behavior

>>> from main import A, B
>>>
>>> As = [A('a'+str(i)) for i in range(10)]
>>> Bs = [B('b'+str(i),As[i]) for i in range(10)]

Deleting an item from the As array removes the item from the list, and thus decrements the refcount, and the subject a1 is deleted, which notifies b1 that the subject is deleted. b1 then calls del on itself, but is not deleted because of the reference in the Bs array.

>>> del As[1]
Subject.__del__
B.deleting, subject=A[name=a1]

Setting the reference in the Bs array to None will reduce refcount to 0, and thus call __del__

>>> Bs[1] = None
B[name=b1].__del__

Showing that the Bs array needs some cleanup, manually

>>> len(As)
9
>>> len(Bs)
10

>>> del Bs[1]
>>> len(Bs)
9

And that the gc doesn't have any work to do:

>>> import gc
>>> gc.collect()
0

The rest get cleaned up on exit()

Answer 2

This is a significant amount of code difference to account for your requirement of automatically maintaining a list of references, and cleaning up when the references are deleted. I've added a Manager class to accomplish this, and a second pass deleted() event that is a hook for the Manager to clean up the lists it is maintaining. I'm re-posting the full, modified code here as it wasn't trivial to update my previous answer.

I believe this fully satisfies the question you've asked. Perhaps not the reason why you need this in the first place, but I think you asked that in the other question.

We need weak references to make this work:

import weakref

The Observer interface gets a new method to be called after deleted()

class Observer(object):
    def update(self,subject,message): pass

    def deleting(self,subject):
        ''' the subject is being deleted '''
        pass

    def deleted(self,subject):
        pass

A manager class maintains the lists of subjects and observers

class Manager(Observer):

    def __init__(self, ):
        self._subjects = []
        self._observers = []

    def monitorSubject(self, subject):
        self._subjects.append( weakref.ref(subject) )

        # observe the subject for deletion to
        # trigger list maintenance on "deleted"
        subject.registerObserver(self)

    def monitorObserver(self, observer):
        self._observers.append( weakref.ref(observer) )

    def deleted(self, subject):
        ''' a subject was deleted, remove it from the list.
        deleting() is called first, and the observers delete themselves.
        deleted() is called next, and is a hook for the manager to
        cleanup any dead subjects and observers '''

        # calling a weakref returns the original object, and `None` when the
        # reference is dead
        def isNotDead(r):
            return not r()==None

        # remove any dead subjects
        print 'Removing dead subjects...'
        self._subjects = filter(isNotDead, self._subjects)

        # remove any dead observers            
        print 'Removing dead observers...'
        self._observers = filter(isNotDead, self._observers, )

    def __str__(self,):
        return "[Manager] Subjects:{0}, Observers:{1}".format(
            ','.join([str(r()) for r in self._subjects]),
            ','.join([str(r()) for r in self._observers])
        )

The differences in the subject are noted in the comments, but primarily there is a second pass to call deleted. The first pass of deleting notifies the observers, while the second pass of deleted notifies the manager. Also, the __del__ routine uses weak refs to iterate over, as some of them get deleted along the way.

class Subject(object):
    def __init__(self):
        self.observers = []

    def __del__(self, ):
        ''' on deletion, notify the observers '''
        print "{0}.__del__".format(self)

        # copy the observer list, then remove all references to the observers
        # NEW - use weakrefs here, or they will not be properly deleted later
        obs = [weakref.ref(o) for o in self.observers]

        # clear all strong references to the observers
        self.observers = []

        # notify all observers that we were deleted
        # ** only if they are not already deleted **
        for o in obs:
            if not o() == None:
                o().deleting(self)

        # NEW - second pass to allow the Manager to cleanup
        # ** only if they are not already deleted **
        for o in obs:
            if not o() == None:
                o().deleted(self)

    def registerObserver(self, observer):
        if observer not in self.observers:
            self.observers.append(observer)

    def removeObserver(self, observer):
        self.observers.remove(observer)

    def notifyObservers(self, message = None):
        for observer in self.observers:
            observer.update(self,message)

Same as before, with simplified string formatting

class A(Subject):
    ''' A is just a subject '''
    def __init__(self, name):
        super(A,self).__init__()
        self.name = name

    def __str__(self):
        return "A[ {0} ]".format(self.name)

B is the same as before

class B(Observer):
    ''' class B is an observer of A '''

    def __init__(self, name, a):
        self.name = name

        # don't keep a reference to 'a', or 'a' will not be deleted!
        a.registerObserver(self)

    def __str__(self):
        return "B[ {0} ]".format(self.name)

    def __del__(self):
        print("{0}.__del__".format(self))

    def deleting(self, subject):
        ''' notification from the subject (A) that it is being deleted. I'm
        assuming here that the subject is actually the one we registered in
        __init__, since I couldn't store a reference or else __del__ would
        not have been called! '''
        print "B[{0}].deleting, subject={1}".format(self.name, subject)

        del self

Some code to execute the file:

if __name__ == '__main__':

    mgr = Manager()

    # keep strong references to the subjects, because
    # we will delete them explicitly
    a1 = A('a1')
    a2 = A('a2')
    mgr.monitorSubject(a1)
    mgr.monitorSubject(a2)

    # monitor observers directly, and do NOT keep
    # strong references or they will not be deleted
    mgr.monitorObserver( B('b1', a1) )
    mgr.monitorObserver( B('b2', a1) )
    mgr.monitorObserver( B('b3', a2) )
    mgr.monitorObserver( B('b4', a2) )

    # show the starting state
    print mgr

    print "Deleting a1..."
    del a1
    print mgr

    print "Deleting a2..."
    del a2
    print mgr

And the output:

    #  OUTPUT (some newlines added)
    #
    #  [Manager] Subjects:A[ a1 ],A[ a2 ], Observers:B[ b1 ],B[ b2 ],B[ b3 ],B[ b4 ]
    #
    #  Deleting a1...
    #  A[ a1 ].__del__
    #  B[ b1 ].__del__
    #  B[ b2 ].__del__
    #  Removing dead subjects...
    #  Removing dead observers...
    #  [Manager] Subjects:A[ a2 ], Observers:B[ b3 ],B[ b4 ]
    #
    #  Deleting a2...
    #  A[ a2 ].__del__
    #  B[ b3 ].__del__
    #  B[ b4 ].__del__
    #  Removing dead subjects...
    #  Removing dead observers...
    #
    #  [Manager] Subjects:, Observers: