2

I have a problem in which there is a python object that is hiding somewhere. The object is a wrapper around a C library and I need to call the deallocation routine at exit, otherwise the (behind the scenes) thread will hang (it's a cython object, so the deallocation is put in the __dealloc__ method).

The problem is I can't for the life of me work out where the object is hiding. I haven't intentionally introduced any global state. Is there some way to work out where an object is lingering? Could it just be a lingering object cycle, so gc should pick it up? That said, I'd really like to work out the cause of the problem if possible.

Edit: I solved the problem, which was down to pyglet event handlers not being cleanly removed. They were in the __del__ method, but the object wasn't being deleted because the event dispatcher had hold of an object method. This is fine logically, but it seems odd to me that the object is never deleted, even at exit. Does anyone know why the __del__ is not called at interpreter exit? Actually, this question has been asked - though the answers aren't brilliant.

Anyway, the basic question still stands - how do I reliably find these lingering references?

Community
  • 1
  • 1
Henry Gomersall
  • 8,434
  • 3
  • 31
  • 54
  • 2
    Have you checked down the back of the couch? :p – Jon Clements May 13 '14 at 10:00
  • @JonClements I actually did the web equivalent - googling it. The only sensible post was me asking pretty much the same question 2 years ago ;) (though that wasn't wholly satisfactory, so here we are). – Henry Gomersall May 13 '14 at 10:02
  • 1
    You probably know more than I do, but here are some ideas. If you get a debug build of Python, you can use the function `sys.getobjects()` to print all currently allocated objects. If it is mostly Cython code, the [Cython debugger](http://docs.cython.org/src/userguide/debugging.html) could probably help. If you know which class is causing the trouble, you could try inserting print statements in the allocation and deallocation methods. Cython's [html annotation](http://docs.cython.org/src/quickstart/cythonize.html#determining-where-to-add-types) may also help clarify things. – IanH May 14 '14 at 16:08
  • Doing something like `print locals(), globals()` might help too. – IanH May 14 '14 at 16:08
  • 1
    @IanH Thanks! The problem is a bit nefarious and makes some interesting suggestions to improve the code. I think there needs to be some clean up code in the Cython module that tidies up at exit, even if the object has not been deleted cleanly. I think Python is far too lax at calling the various delete methods on objects; this is not the first time I've encountered a problem of this type. The lifetime of the objects is a _bit_ too unpredictable. – Henry Gomersall May 14 '14 at 17:30
  • 1
    @HenryGomersall You aren't using any stack frames are you? That is, you aren't using the `introspect` library anywhere? Lingering stack frames can do exactly what you're seeing. – wheaties May 14 '14 at 18:52
  • @wheaties Nope, not that I'm aware of. There is interplay with [pyglet](http://www.pyglet.org/) - the problem occurs when handlers that are not removed explicitly lead to the lingering objects. It _could_ be something to do with how pyglet is controlling the handler; it's not possible to remove method handlers on events somehow attached to the object in the `__del__` method as in that situation, the method is never deleted, despite any `atexit` handlers having been called (and python basically exited - if it weren't for the hardware needing to exit, there would be no indication of a problem). – Henry Gomersall May 15 '14 at 08:32

1 Answers1

0

One possible place is gc.garbage It is a list of objects that have been found unreachable, but cannot be deleted because they include __del__ methods in a cycle.

In Python previous to 3.4, if you have a cycle with several __del__ methods, the interpreter doesn't know in which way they should be executed, as they could have mutual references. So instead it doesn't execute any, and moves the objects to this list.

If you find your object there, the documentation recommends doing del gc.garbage[:].

The solution to avoid this in the first place is to use weakrefs where possible to avoid cycles.

Davidmh
  • 3,797
  • 18
  • 35
  • The problem was it wasn't actually a unreachable object, but an object that I couldn't work out where it was hiding. `gc.garbage` is useful though. ta! – Henry Gomersall May 19 '14 at 15:01