Answering this question about nested pools for multiprocessing:
https://stackoverflow.com/a/40852258/6522112
The code I proposed was not responding the way I wanted. Which is to say if ThreadingPool(8)
and ProcessingPool(3)
, I expected the number of processes to be 24 but I only got 3! So I moved on using this https://stackoverflow.com/a/8963618/6522112 which allows nested process pool by setting the daemon flag to False. This works perfectly but:
- Was it normal to only get 3 processes instead of 24?
- Am I doing it correctly with this hack?
- The final aim would be to do this using several nodes on a cluster. Does this prevent it?
- I am not sure about the type of
map
to consider but it seems that only theimap
combination is doing the job. Any insight?
Here is the modified code that uses pathos
from @MikeMcKerns instead of simple multiprocessing:
import pathos
import multiprocess
class NoDaemonProcess(multiprocess.Process):
"""NoDaemonProcess class.
Inherit from :class:`multiprocessing.Process`.
``daemon`` attribute always returns False.
"""
def _get_daemon(self):
return False
def _set_daemon(self, value):
pass
daemon = property(_get_daemon, _set_daemon)
class NestedPool(pathos.multiprocessing.Pool):
"""NestedPool class.
Inherit from :class:`pathos.multiprocessing.Pool`.
Enable nested process pool.
"""
Process = NoDaemonProcess
Care to be taken if you want to use this, you have to terminate the inner pools otherwise you will end up with zombies:
import NestedPool
def triple(x):
return 3*x
def refork(x):
pool = NestedPool(3)
result = pool.imap(triple, range(5))
result = list(result)
pool.terminate()
return result
pool = NestedPool(8)
result = pool.imap(refork, range(3))