Python 2.7.3 Multiprocessing Pool hanging

Question

I have the following code in ipython where the child process tries to do sys.exit(...) but causes the parent process to hang. Is this a bug? Any idea how to workaround this?

In [1]: from multiprocessing import Pool

In [2]: def f():
   ...:     import sys
   ...:     sys.exit('exiting system...')
   ...:    

In [3]: p = Pool(processes=2)

In [4]: r = p.apply_async(f, [])

In [5]: r.get()   <---- it is hanging here forever.

I have also tried to put raise SystemExit(...) instead of sys.exit(...) but it was the same thing. The only workaround I know is to put raise Exception(...) which turned out to work just fine.

I understand that sys.exit is essentially the same as raise SystemExit, but this exception should be delegated up to its parent process and thus r.get() should be able to receive this exception correct? But it seems to be getting stuck on recv call. Is this a bug in multiprocessing module?

You are not supposted to exit in a call-back function. It has to return instantly. "callback should complete immediately since otherwise the thread which handles the results will get blocked." ( https://docs.python.org/2/library/multiprocessing.html#module-multiprocessing.pool ) — Klaus D., Oct 16 '14 at 10:20
Can you explain why `raise Exception` worked but not `raise SystemExit` in this case? — bohanl, Oct 16 '14 at 10:22
It might be caused by the local import of sys or an special error handing in the ppol class. But that is not the main question. Why do you want to exit in the sub-thread? You should exit in the main thread. If you want to exit the sub-thread, just use return. — Klaus D., Oct 16 '14 at 10:32
@KlausD. The OP isn't using the `callback` keyword arg at all here, so I don't think that quote from the docs is relevant in this case. — dano, Oct 16 '14 at 18:49

score 2 · Accepted Answer · edited May 23 '17 at 12:02

You're causing the Pool worker process to actually exit when you call sys.exit(). The SystemExit exception is special-cased; when you raise it, the process you raised it in exits. The exception doesn't get propagated to the caller. So, that means that in your example, the worker process simply never returns anything back to the parent. And then the parent process will just wait around forever for the child to return something that's never going to get returned. See this question for a more in-depth discussion of that behavior.

I would argue that this is a bug, and that when the sub-process exits, the pool should be marked as broken, and all outstanding tasks should be aborted. This is how concurrent.futures.ProcessPoolExecutor behaves already. I've actually submitted a patch that adds this behavior to multiprocessing.Pool, but so far it hasn't been reviewed.

Now, back to your original question. It looks like you want the parent process to exit here, not just the child. To do that, you'll need to actually return some object back to the parent, and then have the parent exit when it receives that object. It looks like you've discovered you can do this by simply raising an Exception.

Python 2.7.3 Multiprocessing Pool hanging

1 Answers1