1

I have a simulation code in Python that uses much of memory with set/list/dict data structure. The outline is as follows:

massSimulation
    for i in simList:
        individualSimulation

individualSimulation.py
    // do simulation and get the result.
    ... 
    return result

The issue is that it claims memory little by little until it uses more memory (around 12G) than the system can provide (8G) to make the system really slow, the CPU used by python starts 100% then drops very rapidly to almost 0%. If this happens, I kill the python process and start again.

enter image description here enter image description here

I added the garbage reclaim code in the individudalSimulation.py, but the results seem to be the same (I didn't measure, just gut feeling). 

import gc
gc.collect()

What could be a solution to this problems? How can I enforce python to relinquish all the memory it claims when a method is finished?

prosseek
  • 182,215
  • 215
  • 566
  • 871
  • Have a look at [Details how python garbage collection works](http://stackoverflow.com/questions/4484167/details-how-python-garbage-collection-works) – Cilyan Feb 04 '14 at 18:02
  • Did it help, or are you still looking? Maybe the collection is not successful because a lot of variables are still referenced. For example, if you free at the end of the method but did not `del` the variables, they may still be referenced. `gc.collect()` is only useful if you have reference cycles. If this doesn't help, we will need more of your code to investigate. – Cilyan Feb 04 '14 at 18:51
  • Hmm, could a context manager be appropriate here? – Chris Arena Feb 04 '14 at 22:11

1 Answers1

0

Hard to say without seeing more code, but these are my guesses/proposals:

  • If the elements in simList are mutable, and you are adding information on them in the individualSimulation, they can be responsible for your problem since they are still referenced by simList. Avoid that. Even better, use an iterator instead of a list, since all the elements you want to loop through

  • The way you store the results could be using too much space. If there are a lot of simulations, and the results are big, that could be the reason.. you may consider storing them to the hard drive and clean the space in memory.

The point is that you have to delete ALL references to the objects that are taking up so much memory, and then they will be garbage-collectable.

bgusach
  • 14,527
  • 14
  • 51
  • 68