24

I am trying to migrate a bash script to Python.

The bash script runs multiple OS commands in parallel then waits for them to finish before resuming, ie:

command1 &

command2 &

.

commandn &

wait

command

I want to achieve the same using Python subprocess. Is this possible? How can I wait for a subprocess.call command to finish before resuming?

Community
  • 1
  • 1
lisa1987
  • 545
  • 1
  • 5
  • 14
  • possible duplicate of [Multiprocessing vs Threading Python](http://stackoverflow.com/questions/3044580/multiprocessing-vs-threading-python) – sobolevn Jun 06 '15 at 18:56

2 Answers2

33

You can still use Popen which takes the same input parameters as subprocess.call but is more flexible.

subprocess.call: The full function signature is the same as that of the Popen constructor - this functions passes all supplied arguments directly through to that interface.

One difference is that subprocess.call blocks and waits for the subprocess to complete (it is built on top of Popen), whereas Popen doesn't block and consequently allows you to launch other processes in parallel.

Try the following:

from subprocess import Popen
commands = ['command1', 'command2']
procs = [ Popen(i) for i in commands ]
for p in procs:
   p.wait()
Aaron Lelevier
  • 19,850
  • 11
  • 76
  • 111
Martin Konecny
  • 57,827
  • 19
  • 139
  • 159
  • 2
    In your code, command1 and command2 get executed simultaneously? Or command2 gets initiated after command1 is done? – Amir Afianian Apr 09 '18 at 11:10
  • 5
    I've tested, they run in parallel. You can substitute `.wait()` with `.communicate()` and add arguments `stdout=subprocess.PIPE` and `stderr=subprocess.PIPE` to `Popen()` if you want to capture stdout/sterr of the child process instead of letting them clump together with the parent's stdout/stderr – Ferdinand.kraft May 25 '19 at 23:14
  • 2
    They do run in parallel since the subprocesses start when Popen returns. But they block at wait(). So what this code does is, loop once in the array of procs and blocks at the first wait() until it is over. Then blocks at the next wait() if the subprocess is still running. – Yannis Oct 08 '20 at 08:41
  • in case this helps anyone - the elements in `procs` execute upon declaration. so the list is functioning more as an action list than a declaration list. the second for loop goes so quickly that the wait action is initiated probably before the commands execute – rictuar Oct 08 '22 at 04:11
4

Expanding on Aaron and Martin's answer, here is a solution that runs uses subprocess and Popen to run n processes in parallel:

import subprocess
commands = ['cmd1', 'cmd2', 'cmd3', 'cmd4', 'cmd5'] 
n = 2 #the number of parallel processes you want
for j in range(max(int(len(commands)/n), 1)):
    procs = [subprocess.Popen(i, shell=True) for i in commands[j*n: min((j+1)*n, len(commands))] ]
    for p in procs:
        p.wait()

I find this to be useful when using a tool like multiprocessing could cause undesired behavior.

Noah Friedman
  • 143
  • 5