I am using multiprocessing library of python and also emcee which also uses different threads to implement MCMC. The problem is that even when I close the pool still it seems python uses the processors and slows down the cores and I have no idea what is the efficient way to release the cores after the job is done. Could anybody give me an idea of what I should do?
Update:
My code has been already posted here.
Asked
Active
Viewed 1,747 times
0
-
When you say python still uses the processors and slows down the cores, what do you mean? How are you testing this? – FrobberOfBits Aug 26 '14 at 15:23
-
@FrobberOfBits I use ubuntu and then when I run command `top`, I see that my job was not killed and still the cores are being used. – Dalek Aug 26 '14 at 15:24
-
1post your code to give us an idea of exactly how you're closing the pool, also provide more detail about how you're running int. – FrobberOfBits Aug 26 '14 at 15:25
-
@FrobberOfBits I posted the code. – Dalek Aug 26 '14 at 15:32
1 Answers
4
Closing a Pool doesn't stop it from doing work, it just prevents new work items from being added to it:
close()
Prevents any more tasks from being submitted to the pool. Once all the tasks have been completed the worker processes will exit.
So if you have a lot of queued tasks, closing the Pool won't make any difference in resource usage - all the workers will keep consuming those queued tasks until they're gone. If you want to basically abort all the work items immediately, you have to use pool.terminate:
terminate()
Stops the worker processes immediately without completing outstanding work. When the pool object is garbage collected
terminate()will be called immediately.
dano
- 91,354
- 19
- 222
- 219