0

I am using IPython 0.10.2 and Python 2.7 right now. I start one ipcontroller and 20 ipengines on my cluster. The code structure is very simple. I just use MultiEngineClient.execute() methods and MultiEngineClient dictionary interface (e.g., mec['a'] = b) . My current application needs to run nearly two days. However, after 10 hours running, I find that ipcontroller has used 3.1 GB memory. I do use MultiEngineClient.clear_properties(). But it never releases any memory. Does anyone know how to release the memory of ipcontroller ?

Xiao
  • 555
  • 1
  • 5
  • 19

1 Answers1

0

Unfortunately, it may not be possible to free memory effectively in the Controller in IPython 0.10, but the MultiEngineClient.clear_pending_results() method may help.

The controller in 0.11-0.12 can use a database (sqlite or mongodb) to store results, and has been seen to run for a long time, with gigabytes of throughput without unreasonable long-term growth.

minrk
  • 37,545
  • 9
  • 92
  • 87
  • So, for IPython 0.10.2, I can not do anything. – Xiao Jan 11 '12 at 21:47
  • Did you try clear_pending_results() as I mentioned? The clear_properties() function does not affect actual requests/result, so it won't help. – minrk Jan 12 '12 at 22:48
  • My codes does not use unblocked methods. I am not sure if it helps. Any way, I will try it. Thank you! – Xiao Jan 12 '12 at 23:49
  • Hi minrk, I just upgraded to IPython 0.12. For the same question, does IPython 0.12 support explicit memory release without using any database? – Xiao Jan 23 '12 at 20:43
  • Yes, Client.purge_results() allows you to instruct the Hub to forget past results. – minrk Jan 23 '12 at 20:48
  • Even using purge_results does not release any memory. Besides, my client program also consume lots memory too. – Xiao Jan 25 '12 at 22:52
  • Still working on a way to more cleanly keep the local caching down, but for now, see [this note](http://stackoverflow.com/a/8826101/938949) and subsequent comments for a way to eliminate any caching on the controller or client. – minrk Jan 26 '12 at 00:36
  • Now, what I want is the following mechanism: 0. master receives a job 1. master dispatches it to several slave node; 2. slaves run the job 3. master clears all data and references. It is very simple. The hub (master) does need to keep any references after the job is done. This sounds like a MPI program. However, I want my code to running "forever", like a traditional client/server program. – Xiao Jan 26 '12 at 05:06