Python multiprocessing: how to exit cleanly after an error?

Question

I am writing some code that makes use of the multiprocessing module. However, since I am a newbie, what often happens is that some error pops up, putting a halt to the main application.

However, that applications' children still remain running, and I get a long, long list of running pythonw processes in my task manager list.

After an error occurs, what can I do to make sure all the child processes are killed as well?

Answer 1

There are two pieces to this puzzle.

1. How can I detect and kill all the child processes?
2. How can I make a best effort to ensure my code from part 1 is run whenever one process dies?

For part 1, you can use multiprocessing.active_children() to get a list of all the active children and kill them with Process.terminate(). Note the use of Process.terminate() comes with the usual warnings.

from multiprocessing import Process
import multiprocessing

def f(name):
    print 'hello', name
    while True: pass

if __name__ == '__main__':
    for i in xrange(5):
        p = Process(target=f, args=('bob',))
        p.start()

    # At user input, terminate all processes.
    raw_input("Press Enter to terminate: ")
    for p in multiprocessing.active_children():
       p.terminate()

One solution to part 2 is to use sys.excepthook, as described in this answer. Here is a combined example.

from multiprocessing import Process
import multiprocessing
import sys
from  time import sleep

def f(name):
    print 'hello', name
    while True: pass

def myexcepthook(exctype, value, traceback):
    for p in multiprocessing.active_children():
       p.terminate()

if __name__ == '__main__':
    for i in xrange(5):
        p = Process(target=f, args=('bob',))
        p.start()
    sys.excepthook = myexcepthook

    # Sleep for a bit and then force an exception by doing something stupid.
    sleep(1)
    1 / 0