I have the following code (just testing some assumptions about the Pool class):
#!/usr/bin/env python
from time import sleep
from multiprocessing import Pool
def run_process(args):
print "Sleeping %d seconds" % (args * 2)
sleep(args * 2)
print "Slept %d seconds" % (args * 2)
def main():
test = [1,2,3,1,2,3,1,2,3]
pool = Pool()
pool.map_async(run_process, test).get()
if __name__ == '__main__':
main()
Eventually test will become a list of arguments to be passed to each invocation of run_process in order for it to spawn a command (i.e. a separate process).
Now when I press Ctrl+C it unfortunately doesn't even tear down the immediate child processes (from the documentation it sounds like the Pool creates simply child processes). What happens on the terminal (Ubuntu 10.04, Python 2.6.5, if this matters) is that I see the echoed ^C whenever I press it, but neither does the parent die in time when all the children should have exited from the "job" (i.e. sleeping) nor was I able to reclaim the terminal without killing the parent process.
How can I adjust my script so that it will tear down the immediate children (and later on also the spawned commands)? I suppose what I'm trying is to use multiprocessing.Pool.close() and multiprocessing.Pool.terminate() from a signal handler?!
N.B.: I'm intentionally using processes here instead of threads, so I'd appreciate if the solution would take that into account.
Here's one thing I tried after reading through the answers to this question, but it suffers from the same problem:
#!/usr/bin/env python
from signal import signal
from time import sleep
from multiprocessing import Pool
def init_worker():
import signal
signal(signal.SIGINT, signal.SIG_IGN)
def run_process(args):
print "Sleeping %d seconds" % (args * 2)
sleep(args * 2)
print "Slept %d seconds" % (args * 2)
def main():
test = [1,2,3,1,2,3,1,2,3]
try:
pool = Pool()
pool.map_async(run_process, test).get()
except KeyboardInterrupt:
pool.terminate()
pool.wait()
if __name__ == '__main__':
main()
Adding an insanely large timeout (the jobs can run several weeks, so the timeout must be big enough to cover that) to the Pool.get() appears to allow me to use Ctrl+C, although the output is somewhat confusing:
Traceback (most recent call last):
File "./test.py", line 26, in <module>
main()
File "./test.py", line 23, in main
pool.wait()
AttributeError: 'Pool' object has no attribute 'wait'
after getting rid of try/except, the traceback looks "more natural". So the trick is to change the line from the original script pool.map_async(run_process, test).get() to something like pool.map_async(run_process, test).get(999999) where 999999 is a timeout big enough to suit your purpose.
