10

From the docs using Popen.wait() may:

deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.

In communicate docs it's written that:

The data read is buffered in memory, so do not use this method if the data size is large or unlimited

How to reproduce such a problematic behavior and see that using Popen.communicate() fixes it?

Deadlock means some circular wait between processes holding resources happens and is stuck forever. What is the circular dependency here? Python process which waits for the child process to terminate is one wait. What's the other? Who waits for what in the below scenario?

it blocks waiting for the OS pipe buffer to accept more data

rok
  • 9,403
  • 17
  • 70
  • 126
  • 1
    If you reached the deadlock, communicate won't 'fix' it. You should use communicate in the first place, which avoids reaching the deadlock. – Chen A. Apr 09 '18 at 08:25
  • actually i had a situation when the code using wait was stuck each time it was run, until it was modified to use communicate and it wasn't stuck anymore. so i want to reproduce the case of deadlock and see that communicate solves it – rok Apr 09 '18 at 13:54

1 Answers1

10

It's easy.

Create a process which outputs a lot of text and don't read the output:

p = subprocess.Popen(["ls","-R"],stdout=subprocess.PIPE)
p.wait()

after a while the standard output pipe is full and process is blocked.

It's a deadlock situation because the subprocess cannot write anymore to the output until it's consumed (which is: never), and the python process waits for the subprocess to finish.

To avoid the deadlock, you could use a read line loop:

p = subprocess.Popen(["ls","-R"],stdout=subprocess.PIPE)
for line in p.stdout:
    # do something with the line
p.wait()

communicate also fixes that but also fixes the much trickier case where both output and error streams are redirected to separate streams (in that case, the naive loop above could still deadlock).

Let's suppose you have a compilation process

p = subprocess.Popen(["gcc","-c"]+mega_list_of_files,stdout=subprocess.PIPE,stderr=subprocess.PIPE)

Now you want to get the output from this one, so you do:

output = p.stdout.read()

unfortunately, a lot of errors pop up instead, blocking the error stream while you're reading the output stream: deadlock again.

Try to read error stream instead, and the exact opposite could occur: lots of stdout output blocking your process.

communicate uses multithreading to be able to process output & error streams at the same time and keep them separated, without risk of blocking. Only caveat is that you cannot control the process output line by line / print program output in real time:

p = subprocess.Popen(["gcc","-c"]+mega_list_of_files,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
output,error = p.communicate()
return_code = p.wait()
Jean-François Fabre
  • 137,073
  • 23
  • 153
  • 219
  • 1
    thanks. what entity should consume the output that's full? parent process? don't understand also how communicate fixes it. it's written in its docs that `do not use this method if the data size is large or unlimited` – rok Apr 11 '18 at 12:46
  • 2
    yes, your python process. Of course if the output is unlimited, communicate never ends & consumes all the memory. in that case you have to kill the process when encountering a given line for instance. You have to read it line by line in that case. – Jean-François Fabre Apr 11 '18 at 12:56
  • 1
    _Great_ answer. Seems like there's an idea that you must always use `communicate()` when using `PIPE`, which as you point out isn't the case. The docs aren't actually wrong, in that the read line loop prevents _and the child process generates enough output to a pipe such that it blocks..._. – user1071847 Nov 29 '19 at 16:24
  • if you use `communicate()` there are things you cannot do, like reading line by line and performing actions in real-time when the process is outputting data. Simplest example: printing and logging to the console at the same time, and detecting a special line to terminate the process (specially if the process runs forever if not stopped, which forbids the use of communicate…). Also most of the time merging output & error streams are okay, and it avoids deadlocks (if output is consumed of course) – Jean-François Fabre Nov 29 '19 at 20:15