121

I have a main process which creates two or more sub processes, I want main process to wait until all sub processes finish their operations and exits?

 # main_script.py

 p1 = subprocess.Popen(['python script1.py']) 
 p2 = subprocess.Popen(['python script2.py'])
 ... 
 #wait main process until both p1, p2 finish
 ...
Nikhil Rupanawar
  • 4,061
  • 10
  • 35
  • 51
  • 6
    use the wait method: `p1.wait(); p2.wait()` – catalin.costache Feb 27 '13 at 08:52
  • 1
    check this question out: http://stackoverflow.com/questions/6341358/subprocess-wait-not-waiting-for-popen-process-to-finish-when-using-threads – Calum Feb 27 '13 at 08:54
  • 2
    http://stackoverflow.com/questions/100624/python-on-windows-how-to-wait-for-multiple-child-processes – Gjordis Feb 27 '13 at 09:00
  • 2
    BTW, `Popen(['python script1.py'])` won't work. Either do `Popen(['python', 'script1.py'])` (to be preferred) or `Popen('python script1.py', shell=True)`. – glglgl Feb 27 '13 at 09:08
  • 2
    Not a duplicate--linked duplicate is specifically regarding Windows. – Will May 13 '16 at 00:34

2 Answers2

156

A Popen object has a .wait() method exactly defined for this: to wait for the completion of a given subprocess (and, besides, for retuning its exit status).

If you use this method, you'll prevent that the process zombies are lying around for too long.

(Alternatively, you can use subprocess.call() or subprocess.check_call() for calling and waiting. If you don't need IO with the process, that might be enough. But probably this is not an option, because your if the two subprocesses seem to be supposed to run in parallel, which they won't with (call()/check_call().)

If you have several subprocesses to wait for, you can do

exit_codes = [p.wait() for p in p1, p2]

(or maybe exit_codes = [p.wait() for p in (p1, p2)] for syntactical reasons)

which returns as soon as all subprocesses have finished. You then have a list of return codes which you maybe can evaluate.

glglgl
  • 89,107
  • 13
  • 149
  • 217
  • 1
    If I have a list of processes that I need to wait for, but I want the wait to be interrupted the moment any of the processes has finished (so I can later resume it to wait for the remaining processes), how would I go about that? – antred Jun 01 '16 at 10:48
  • 1
    @antred Then you should call `.wait()` with a (rather small) timeout and process the results. AFAIR, it returns `None` if the said process wasn't terminated yet and a number if it was. – glglgl Jun 01 '16 at 10:50
  • 1
    @glglgl Yeah, that would work, though I wanted to avoid having to actively poll, but I guess there may not be a way around it (I'm on a Windows machine). Also, apparently the same thing was asked and answered here: http://stackoverflow.com/questions/100624/python-on-windows-how-to-wait-for-multiple-child-processes – antred Jun 01 '16 at 10:52
  • @antred Of course - you can have a thread wait on each process and tell as soon as it finished. That would be a solution I didn't think of. It would lead to maximum responsiveness. – glglgl Jun 01 '16 at 10:54
  • 1
    @glglgl I wish Python on Windows would just emulate what os.waitpid( 0 ) does on Unix. Even if, under the covers, the emulation would have to do the same thing the threaded solutions in that other SO post do, it would still be nice to have Python offer this functionality out-of-the-box. – antred Jun 01 '16 at 10:59
  • @glglgl Thanks soo much for explaining the difference between `subprocess.call` and `subprocess.Popen` . It is really helpful to know that only the latter shows IO and runs in parallel! – Agile Bean Aug 02 '18 at 06:12
  • tried with Python 3.8.8, this method does not work anymore. – Franva Apr 30 '22 at 09:03
  • @Franva I didn't know "does not work" is a valid way of describing an error. But maybe the edit I made to my answer solves your problem. – glglgl May 02 '22 at 06:34
42
subprocess.call

Automatically waits , you can also use:

p1.wait()
Gjordis
  • 2,540
  • 1
  • 22
  • 32