91

I'm completely confused between subprocess.call() , subprocess.Popen(), subprocess.check_call().

Which is blocking and which is not ?

What I mean to say is if I use subprocess.Popen() whether the parent process waits for the child process to return/exit before it keep on its execution.

How does shell=True affect these calls?

Tomerikoo
  • 18,379
  • 16
  • 47
  • 61
Kumar Roshan Mehta
  • 3,078
  • 2
  • 27
  • 50
  • did you try `subprocess.call([cmd,params,&])`? – Charlie Parker Oct 24 '17 at 19:16
  • @CharlieParker the `&` would have to be in quotation marks, and the `params` would have to be separate elements of the list, and it would have to use `shell=True`, and actually it should actually use a string instead of a pre-parsed list. It does work, but it's a hack. – Karl Knechtel May 08 '23 at 14:11

1 Answers1

126

Popen is nonblocking. call and check_call are blocking. You can make the Popen instance block by calling its wait or communicate method.

If you look in the source code, you'll see call calls Popen(...).wait(), which is why it is blocking. check_call calls call, which is why it blocks as well.

Strictly speaking, shell=True is orthogonal to the issue of blocking. However, shell=True causes Python to exec a shell and then run the command in the shell. If you use a blocking call, the call will return when the shell finishes. Since the shell may spawn a subprocess to run the command, the shell may finish before the spawned subprocess. For example,

import subprocess
import time

proc = subprocess.Popen('ls -lRa /', shell=True)
time.sleep(3)
proc.terminate()
proc.wait()

Here two processes are spawned: Popen spawns one subprocess running the shell. The shell in turn spawns a subprocess running ls. proc.terminate() kills the shell, but the subprocess running ls remains. (That is manifested by copious output, even after the python script has ended. Be prepared to kill the ls with pkill ls.)

unutbu
  • 842,883
  • 184
  • 1,785
  • 1,677
  • 6
    [`start_new_session=True` or its analogs](http://stackoverflow.com/a/13256908/4279) enables terminating the whole process tree at once. See [How to terminate a python subprocess launched with shell=True](http://stackoverflow.com/q/4789837/4279) – jfs Feb 21 '14 at 14:33
  • Thank you your answer helped me a lot. P.S. `proc = subprocess.Popen(['/run_python.py', '-n', '10'], shell=True)` is different from that `proc = subprocess.Popen("run_python.py -n 10", shell=True)` Only the last one works if the command is built on top of a python script. – dotslash Apr 01 '16 at 20:42
  • 2
    @dotslash: `shell=True` can be a [security hazard](https://docs.python.org/2/library/subprocess.html#using-the-subprocess-module) if the command comes from user input. So it is better to use `shell=False` if possible. With `shell=False`, supply the command and its arguments in a list: `proc = subprocess.Popen(['/run_python.py', '-n', '10'], shell=False)`. – unutbu Apr 01 '16 at 20:48
  • @unutbu so `proc = subprocess.Popen(['/run_python.py', '-n', '10'], shell=False)` is this a blocking process? I `run_python.py` run at background which may last for 1 hour. – dotslash Apr 01 '16 at 20:52
  • 2
    @dotslash: `Popen` is always non-blocking. It is calling `wait` or `communicate` which blocks. – unutbu Apr 01 '16 at 20:59
  • But if I don't use `wait` my command [never executes](http://stackoverflow.com/questions/38012773/pygtk-progressbar-to-display-something-before-subprocess-starts)..? – yPhil Jun 24 '16 at 13:07
  • I don't see this behaviour on Windows 10 with Python 3. I am running from VSCode's integrated console. I also tried on the default terminal. All child processes are killed! – coder.in.me Sep 28 '16 at 15:26
  • 1
    what is wrong with using `subprocess.call([cmd,params,&])`? i.e. adding the standard `&`? – Charlie Parker Oct 24 '17 at 19:17
  • 3
    so if i would like to use `shell=True` and i would like it to block the execution until the inner process of the shell ends, how could i do it? – Yuval Zilber Dec 27 '21 at 12:07
  • 1
    @YuvalZilber simply *don't* include `&` in the shelled command. If the underlying process isn't backgrounded, the shell process will block on it, and `subprocess.call` in Python will block on the shell process. – Karl Knechtel May 08 '23 at 14:13
  • The last example does not make sense. The shell process terminates immediately, as evidenced by the fact that the subsequent Python code runs immediately. The `proc.terminate` etc. is superfluous. An easier way to test, for people with Git installed, is `subprocess.call('gitk', shell=True)` at the REPL; the REPL will hang until the Gitk window is dismissed, because Python is blocking on the shell, and the shell is blocking on Gitk. However, with `subprocess.call('gitk &', shell=True)`, the repl will not hang, because the shell does not block on Gitk, because the Gitk process was backgrounded. – Karl Knechtel May 08 '23 at 14:19