Python Subprocess wait for a list of subprocess

Question

I am using python3 subprocess module to cope with bash script call. I want to control the number of concurrent subprocess when calling the script.

Popen.wait(timeout=None) Wait for child process to terminate. Set and return returncode attribute.

I know that I can call the subprocess.Popen.wait(timeout=None) to wait for child process to terminate. But I was wondering about if I can wait for a list of subprocess.Popen to finish, with that, I can control the number of concurrent processes.

Sample code snippets as following:

Example.py

import subprocess
from itertools import zip_longest

seed = [1, 2, 4, 1, 3, 5, 4, 1]
basepath="/path/to/file/hello.sh"

def grouper(iterable, n, fillvalue=None):
    args = [iter(iterable)] * n
    return zip_longest(*args, fillvalue=fillvalue)

def bash_handler(num):
    return subprocess.Popen('bash {0} {1}'.format(basepath, num), shell=True)

for bulk in grouper(seed, 2):
    [bash_handler(item).wait() for item in bulk if item is not None]
# This will executed one by one

hello.sh

# Validation goes here.
echo $1
sleep $1
echo "Finish Executed"

@MaheshKaria No idea how can I use it. Any help or helpful threads? — shady, Dec 20 '17 at 05:50
Oops, I didn't really mean to close as duplicate (didn't realize this was also tagged [tag:bash]). I'll be happy to reopen if the current duplicate doesn't solve your problem, or at least look for a different duplicate. — tripleee, Dec 20 '17 at 08:01

score 0 · Answer 1 · answered Dec 20 '17 at 07:09

0

when you use communicate(), you don't need to specify the wait for the subprocess to exit. more information can be found here https://docs.python.org/2/library/subprocess.html other link Understanding Popen.communicate

below changes should work for you

from subprocess import Popen,PIPE
def bash_handler(num):
    return subprocess.Popen('bash {0} {1}'.format(basepath, num), shell=True,stdout=PIPE)


for bulk in grouper(seed, 2):
    [bash_handler(item).communicate() for item in bulk if item is not None]

answered Dec 20 '17 at 07:09

pankaj mishra

2,555
2
17
31

Running `bash` with `shell=True` is ... wacky. See also https://stackoverflow.com/questions/3172470/actual-meaning-of-shell-true-in-subprocess – tripleee Dec 20 '17 at 07:28
@pankaj mishra Notice that I want the subprocess in `bulk` to run simultaneously. Is there any way to achieve this? Or I just need to switch to the multiprocessing module? – shady Dec 20 '17 at 07:39
@tripleee could be the right person to answer this – pankaj mishra Dec 20 '17 at 07:56
`Popen` by default runs separate processes, they will essentially churn on in the background if you don't need to `communicate()` specifically with them. Use `subprocess.poll()` to examine the status without blocking. – tripleee Dec 20 '17 at 07:59

Python Subprocess wait for a list of subprocess

1 Answers1