Join a group of python processes with a timeout

Question

I'm using python's multiprocessing library to create several processes.

from multiprocessing import Process
processes = [Process(target=function) for function in FUNCTIONS]
for p in processes:
    p.start()

I want them to run for some duration and then if they have not completed, terminate them.

DURATION = 3600

A bad way to do it is as follows (bad because if the processes finish faster than DURATION, it still waits for all of DURATION):

from time import sleep
sleep(duration)
for p in processes:
    p.join(0)
    p.terminate()

Another bad way to do it (bad because it can possibly take N * DURATION to finish, where N is the number of processes):

for p in processes:
    p.join(DURATION)
    p.terminate()

What is a good way to do this?

I don't know what your processes are doing but normally you would use a signalling variable to signal to the process it's time to finish. So you would way DURATION, set the var and wait to join. — RedX, May 15 '14 at 20:22

wubwubb · Accepted Answer · 2014-05-19T22:02:57.780

2

I believe this does what you want without any polling required, and it will only wait up to your specified DURATION.

time_waited = 0
then = time.time()
for p in processes:
    if time_waited >= DURATION:
        p.join(0)
        p.terminate()
    p.join(DURATION - time_waited)
    time_waited = time.time() - then

edited May 19 '14 at 22:02

answered May 15 '14 at 19:39

wubwubb

325
2
5

Why do you update `then`? You can just have `time_waited = now - then` instead of the last two lines. – Zags May 16 '14 at 22:18
Hmm yea that'd work fine too. I guess for some reason the way I have it was what naturally came to me first. Maybe your way is a bit clearer though. I'll change it. – wubwubb May 19 '14 at 22:01

Zags · Answer 2 · 2014-05-15T20:19:59.980

This will poll every second for whether all of the processes are completed, up to DURATION. If either all processes have finished or DURATION has occurred, it joins/kills all of the processes. It's not perfect, in that it will take slightly longer than a second in each iteration of the for loop, but it will be pretty close.

from time import sleep
for _ in range(DURATION):
    if not any(i.is_alive() for i in processes): break
    sleep(1)
for p in processes:
    p.join(0)
    p.terminate()

Sneftel · Answer 3 · 2014-05-16T14:56:21.057

0

The easiest thing to do is probably to have a "master" thread which join(0)s all the processes and then exits, and have the main thread join(3600) the master thread.

def wait_func():
    for p in processes:
    p.join()

wait_process = Process(target=wait_func)
wait_process.start()
wait_process.join(DURATION)
for p in processes:
    p.terminate()

edited May 16 '14 at 14:56

answered May 15 '14 at 20:10

Sneftel

40,271
12
71
104

Join a group of python processes with a timeout

3 Answers3

Linked