Python3 - Run subprocess commands in for loop parallelly without intrupting one another

Question

I need to run subprocess commands in for loop parallelly without interrupting one another. I have more than 100 shell commands. Some run for a short period and some take time. I don't want to wait for long-running commands. Below are the example. "cmds" is a list of commands

for cmd in cmds:
    push=subprocess.Popen(cmd, shell=True,stdout = subprocess.PIPE)
    push.wait()
    print(push.communicate()[0])

use a [process pool](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool) — Jan Wilamowski, Jul 08 '21 at 06:20
Does this answer your question? [How do I parallelize a simple Python loop?](https://stackoverflow.com/questions/9786102/how-do-i-parallelize-a-simple-python-loop) — Jan Wilamowski, Jul 08 '21 at 06:21
How can I call subprocess through process pool? No of commands can vary in list — Shubham Sharma, Jul 08 '21 at 06:22
for the process pool, I have to define a number of processes but here a number of commands always vary. If possible can you give one example of how to do it — Shubham Sharma, Jul 08 '21 at 08:20

score 2 · Accepted Answer · answered Jul 08 '21 at 08:31

Use a process pool, specify how many processes should run in parallel and let the pool handle the job scheduling:

from multiprocessing import Pool

def run_command(cmd):
    push=subprocess.Popen(cmd, shell=True,stdout = subprocess.PIPE)
    push.wait()
    return push.communicate()[0]

pool = Pool(processes=8)
results = pool.map(run_command, cmds)

for result in results:
    print(result)

score 0 · Answer 2 · answered Jul 08 '21 at 08:33

Through this we can run multiple commands in async way

import subprocess
import asyncio
import time

def background(f):
    def wrapped(*args, **kwargs):
        return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs)

    return wrapped

@background
def func1():
    cmd = 'echo "fun1 1"; sleep 5; echo "func1 2"; sleep 5; echo "func1 3"'
    subprocess.call(cmd, shell=True)

@background
def func2():
    cmd = 'echo "fun2 1"; sleep 5; echo "func2 2"; sleep 5; echo "func2 3"'
    subprocess.call(cmd, shell=True)


func1()
func2()

print('loop finished')

Python3 - Run subprocess commands in for loop parallelly without intrupting one another

2 Answers2