Python3 parallel processes with their own timeouts

Question

I have a requirement where I have to launch multiple applications, get each of their stdout,stderr and these applications run infinitely. The processes do not exchange/share data and are independent of each other. To do variable stress tests, their timeouts might be different.

For eg: 
app1 -> 70sec
app2 -> 30sec
.
.
appN -> 20sec

If these were multiple apps with one common timeout, I would have wrapped it in a timed while loop and killed all processes in the end.

Here are some approaches I think should work:

A timer thread for each app, which reads stdout and as soon as it expires, kills the process. The process is launched within the thread
One timer thread that loops through a dictionary of pid/process_objects:end_time for each process and kills the process when its end_time is >= current time

I have tried using asyncio gather, but it doesn't fully meet my needs and I have faced some issues on Windows.

Are there any other approaches that I can use?

score 2 · Answer 1 · answered Mar 27 '20 at 16:59

2

Second option is pretty production-ready. Have a control loop where you poll for processes to complete and kill them if they are timing out

answered Mar 27 '20 at 16:59

Maks

1,527
1
13
16

score 0 · Accepted Answer · answered Apr 07 '20 at 20:27

Here is the code for the second approach (extended https://stackoverflow.com/a/9745864/286990).

#!/usr/bin/env python
import io
import os
import sys
from subprocess import Popen
import threading
import time
import psutil

def proc_monitor_thread(proc_dict):
    while proc_dict != {}:
        for k,v in list(proc_dict.items()):
            if time.time() > v:
                print("killing " + str(k))
                m = psutil.Process(k)
                m.kill()
                del proc_dict[k]
        time.sleep(2)

pros = {} 

ON_POSIX = 'posix' in sys.builtin_module_names

# create a pipe to get data
input_fd, output_fd = os.pipe()

# start several subprocesses
st_time = time.time()
for i in ["www.google.com", "www.amd.com", "www.wix.com"]:
    proc = Popen(["ping", "-t", str(i)], stdout=output_fd,
                   close_fds=ON_POSIX) # close input_fd in children

    if "google" in i:
        pros[proc.pid] = time.time() + 5
    elif "amd" in i:
        pros[proc.pid] = time.time() + 8
    else:
        pros[proc.pid] = time.time() + 10

os.close(output_fd)
x = threading.Thread(target=proc_monitor_thread, args=(pros,))
x.start()

# read output line by line as soon as it is available
with io.open(input_fd, 'r', buffering=1) as file:
    for line in file:
        print(line, end='')
#

print("End")
x.join()

Python3 parallel processes with their own timeouts

2 Answers2