How to terminate an AsyncResult from multiprocessing pool?

Question

I have a simple code like the one shown below. The first process holds back the queue, so none of them are completed.

I would like to be able to kill an AsyncResult if it exceeds the .get() timeout, so my pool queue can move on. However I couldn't find any simple way to do it, without modifying "myfunc". Does anyone have an idea how this could be achieved?

import multiprocessing
import time


def myf(x):
    if x == 0:
        time.sleep(100)
    else:
        time.sleep(2)
    return 'done'


pool = multiprocessing.Pool(processes=1)
results = []
for x in range(8):        
  results.append(pool.apply_async(myf,args=[x]))
pool.close()

for res in results:
    try:
      print res.get(3)
    except Exception as e:
      print 'time out'

Your code will "move on" in 3 seconds per blocked process. Are you saying you want to cancel `myf` itself? — tdelaney, Feb 23 '16 at 18:32
I understand that the for loop will move on, but I would the pool queue to move on as well. For instance, I would like to terminate "myf" so the next pool process would start. — user1477337, Feb 23 '16 at 18:54
you could take a look at this: http://stackoverflow.com/questions/492519/timeout-on-a-python-function-call but from looking at the `multiprocessing.pool` source code if I correctly understood the `worker` function there is no way to do it without changing `myf` or wrapping it in some kind of alarm. — Tadhg McDonald-Jensen, Feb 23 '16 at 19:54

score 5 · Accepted Answer · edited May 23 '17 at 12:07

multiprocessing.Pool has not been designed for such use case.

Forcing one of the workers to commit suicide will lead to undefined behaviour which might vary from remaining stuck there forever to getting your program to crash.

There are libraries which can solve your problem. pebble allows you to set timeout to your workers and will stop them if the time limit has exceeded.

How to terminate an AsyncResult from multiprocessing pool?

1 Answers1