How to catch exceptions in workers in Multiprocessing

Question

I'm working with the multiprocessing module in Python (2.7.3) and want to debug some stuff going on in my workers. However, I seem to not be able to catch any exceptions in the worker threads.

A minimal example:

import multiprocessing as mp


a=[1]
def worker():
    print a[2]


def pool():
    pool = mp.Pool(processes=1)
    pool.apply_async(worker, args = ())
    pool.close()
    pool.join()
    print "Multiprocessing done!"


if __name__ == '__main__':
    pool()

This is expected to raise an IndexError, but my output only is

    Multiprocessing done!

Is there a way to show me all exceptions occuring in the worker threads without manually raising my own?

falsetru · Accepted Answer · 2014-06-21T13:07:23.297

31

The error is not raised unless you call get method of AsyncResult (the return value of the apply_async):

According to the AsyncResult.get documentation:

Return the result when it arrives. If timeout is not None and the result does not arrive within timeout seconds then multiprocessing.TimeoutError is raised. If the remote call raised an exception then that exception will be reraised by get().

def pool():
    pool = mp.Pool(processes=1)
    result = pool.apply_async(worker, args=())
    result.get() # <------------
    pool.close()
    pool.join()
    print "Multiprocessing done!"

edited Jun 21 '14 at 13:07

answered Feb 28 '14 at 12:14

falsetru

357,413
63
732
636

Wow, THAT was fast. Thanks. – Dschoni Feb 28 '14 at 12:21
Sorry, I guess that was me by accident. I come back to this answer so often...However my downvote can't be canceled for 2 hours. Sorry. – Dschoni Jun 20 '14 at 14:39
1

If I don't want to wait for the process to finish and still wants to get exceptions, what can I do? I mean using `result.get()` would block the code until the process is finished how to escape that? – Masoud Rahimi Apr 08 '19 at 08:06
AFAIK, exception -> cause the process termination. You need to wait the process end. – falsetru Apr 08 '19 at 15:23

score 3 · Answer 2 · edited May 23 '17 at 11:54

I think falsetru gave you what you need. I'd just like to expand a little more.

If it's important for you to get not only the error but the original context (i.e. to know that the exception occurred on line 1 of worker()) then you can check this nice post by Ned Batchelder which explains how to reraise exceptions with their original context.

That doesn't work for mp.Pool so it's just in case you need something more. This SO Question covers your question using more explicit multiprocessing techniques instead of mp.Pool.

How to catch exceptions in workers in Multiprocessing

2 Answers2

Linked