4

Basically, I have a long running process where I would like to be able to unimport modules and recover memory via the gc. I've read about deleting modules How do I unload (reload) a Python module? and it seems like there are still dangling references that block gc.

However, what if I import and use the module only inside a namespace. In other words, something like this:

ns = {}
exec somecode in ns

Then I would cleanup sys.modules inside the namespace and finish off by deleting the namespace itself.

Would that free up the memory for reuse in CPython?

If not, then is it possible to access some part of the Python C API using ctypes, to accomplish this?

The important part of the end result is that memory is released so that a process running for weeks or months, can reliably unimport a module without reloading it. Of course it is entirely possible that any given module would be loaded and unloaded many times during that time period. I am assuming that a module could create a large number of objects while it is loaded, and that the normally cleanup (sys.modules and del) would leave those objects in memory forever.

Jochen: Yes, I could work around this in a number of ways but I am interested in exploring the limits of Python.

Community
  • 1
  • 1
Michael Dillon
  • 31,973
  • 6
  • 70
  • 106
  • 2
    Are you sure you can't work around this by moving the relevant functionality into a self-contained function run with [multiprocessing](http://docs.python.org/library/multiprocessing.html) or [subprocess](http://docs.python.org/library/subprocess.html)? – Nicholas Knight May 31 '11 at 01:33
  • 3
    Sounds like a lot of work to save just a few bytes. Do you import thousands of modules, or why do you think unloading them would be worth it? – Jochen Ritzel May 31 '11 at 01:34

2 Answers2

3

If what you really want is to avoid memory leaks, Your best bet is probably to arrange for importing the module once, in the normal way, with sys.modules in it's usual state. No matter how many times the module is later imported, it will not take any more memory, since the import machinery will just keep returning the same module.

If for some reason, this still doesn't suit, say modules are being created dynamically and only need to be used once, exec certainly isn't the solution. You should consider using an alternative execution model, perhaps forking new processes.

SingleNegationElimination
  • 151,563
  • 33
  • 264
  • 304
3

To un-import a module you will need to ensure that you've removed all references to the module. That means you have to delete the references from all modules that imported it, delete the reference from sys.modules, delete any references to any functions or classes defined in that module and delete all references to objects that are instances of classes defined in the module.

In almost all situations this is more effort than it is worth to retrieve what is a comparatively small amount of memory. If you really want to try this then gc.get_referrers() might be useful as you can delete all but one known reference the the module and then trace back to find what else still references it.

Duncan
  • 92,073
  • 11
  • 122
  • 156