0

I have a function that uses multiprocessing.pool to perform some heavy computation (modeled with sleep(5)), and I want to have a timeout that ends this computation if it takes too long, and closes the pool beforehand.

def add_const(number, const):
    print('Adding', const,'to', number)
    time.sleep(5)
    return number+const
def function_parallel(list_of_numbers, const):
    pool = Pool(2)
    data=pool.imap(functools.partial(add_const, const=const), list_of_numbers)
    pool.close()
    pool.join()
    return(data)

I tried using stopit module (https://pypi.org/project/stopit/), but it didn't work.

t1 = time.time()
with stopit.ThreadingTimeout(2) as to_ctx_mgr:
    assert to_ctx_mgr.state == to_ctx_mgr.EXECUTING
    result=[]
    data=function_parallel([1, 2], 2)
    for i in data:
        result.append(i)
if to_ctx_mgr.state == to_ctx_mgr.TIMED_OUT:
    result.append('timeout')
print(result)
print(time.time()-t1)

With map in function_parallel it took full 5 seconds instead of stopping after 2, and it also left zombie processes. With imap computation freezed and had to be forced to stop, so even worse.

Is there a safe and reliable way to timeout this function? Thanks!

Edit:

Unfortunately I couldn't apply pebble to my problem. It allows to set timeout for each call of add_const and I need to have one timeout for the whole pool (like if there's a million calls, each one is quite short, but together they take a lot of time).

Even more, in general I would like to timeout some very big part of the code including pool (sometimes timeout can even happen before the pool), but it seems like there is no solution for that...

Anna
  • 199
  • 1
  • 10
  • Relevant [pool-with-timeout-on-each-process-not-all-of-the-pool](https://stackoverflow.com/questions/31255118) and [pool-timeout](https://stackoverflow.com/questions/38711840) – stovfl Feb 26 '20 at 14:38
  • 1
    Does this answer your question? [Multiprocessing - map over list, killing processes that stall above timeout limit](https://stackoverflow.com/questions/44402085/multiprocessing-map-over-list-killing-processes-that-stall-above-timeout-limi) – noxdafox Feb 26 '20 at 14:40
  • 1
    Other duplicates: https://stackoverflow.com/questions/54066993/how-to-get-precise-timeout-with-python-multiprocess-apply-async/54081814#54081814 https://stackoverflow.com/questions/37098360/creating-a-timeout-function-in-python-with-multiprocessing/37098838#37098838 https://stackoverflow.com/questions/54011552/any-concurrent-futures-timeout-that-actually-works/54012069#54012069 – noxdafox Feb 26 '20 at 14:45

0 Answers0