Handling per worker timeout with Python multiprocessing

Question

I've been having some troubles with Python's multiprocessing module. I need to test a function with different parameters, and as that function does a lot of calculations, the use of all the cores is most desirable. I ended up using pool.map(), which suits my needs. The problem is that sometimes my function never ends, so pool.map keeps blocked forever expecting a returned value. I don't know why this happens. I've done a lot of tests without using multiprocessing, just passing an argumment after another in a for loop, and it always ends.

Anyway, What I want to do now is to specify a timeout for each worker / execution of the function, but I need a variable inside the function to be returned in case that timeout is reached. That would be like the status of the function before the timeout happens. My code is too big to post it here, but here's a simple, equivalent example:

def func(x):
    secsPassed = 0
    for _ in xrange(x):
        time.sleep(1)
        secsPassed +=1

    return secsPassed

pool = Pool(4)
results = pool.map(func, [3, 10, 50, 20, 300])

So I'd like that each execution takes at max 30 seconds, and I'd also like to know the value of secsPassed just before func gets interrumpted. I'm using Python 2.7 and I can make changes to func, or use another tool aside from Pool.map if necessary.

Thanks in advance.

Maybe change to celery? It allows timeouts, and retries easily. Plus if you setup RabbitMQ on a server, you could use multiple servers or PC by deploying your code that connects to the same RabbitMQ. — user1157751, Feb 01 '16 at 18:09
I'll check that out, but I'd prefer to avoid using third party libraries for this project. — Sebnarvaez, Feb 01 '16 at 18:38

score 0 · Answer 1 · edited May 23 '17 at 11:45

This question has been asked several times in the past.

multiprocessing.Pool has not been designed for such use case.

Forcing one of the workers to commit suicide will lead to undefined behaviour which might vary from remaining stuck there forever to getting your program to crash.

There are libraries which can solve your problem. pebble allows you to set timeout to your workers and will stop them if the time limit has exceeded.

Handling per worker timeout with Python multiprocessing

1 Answers1