subprocess.Popen and buffered process output

Question

From inside python code, I want to run a binary program that gets its parameters from stdin. Using the subprocess module, this should be straightforward:

import subprocess
command = [ 'my_program' ]
p = subprocess.Popen( command,  \
        stdin = subprocess.PIPE, stdout = subprocess.PIPE, \
        env={ "GFORTRAN_UNBUFFERED_ALL": "1"} )
p.stdin.write ( stdin_stuff )
while True:
  o = p.stdout.readline()
  if p.poll() != None: 
    break
  # Do something with stdout

Now, this launches the program, but the python script just hangs there. I understand that this may well be due gfortran (which I use to compile my_program is buffering its stdout stream. gfortran allows one to use the GFORTRAN_UNBUFFERED_ALL environmental variable, as I have done, as well as using the FLUSH() intrinsic in the fortran code, but still no luck: the python code still hangs.

You can accept an answer you like by "checking" the mark next to it. It is considered polite on this site to do so, so you might want to check the answers you like best on the questions you have asked. — Sven Marnach, Feb 09 '11 at 23:31

score 4 · Answer 1 · answered Feb 09 '11 at 16:42

4

You should have better luck using Popen.communicate() to send strings to the process' stdin rather than manually writing to it.

stdoutdata, stderrdata = p.communicate(stdin_stuff)

answered Feb 09 '11 at 16:42

Aphex

7,390
5
33
54

score 2 · Answer 2 · edited Jun 20 '20 at 09:12

2

To complement Aphex's answer, here the relevant part of the documentation:

Warning

Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.

edited Jun 20 '20 at 09:12

Community

1
1

answered Feb 09 '11 at 16:50

Sven Marnach

574,206
118
941
841

This might be better served as an update to @Aphex or a comment. It doesn't seem like a good answer all by itself like this. – S.Lott Feb 09 '11 at 17:02
@S.Lott: Thought about making it a comment, but the formatting really doesn't work as a comment. That's why I explicitly stated it's a complement to Aphex's answer. – Sven Marnach Feb 09 '11 at 17:07
"the formatting really doesn't work as a comment". True. Why not an update? – S.Lott Feb 09 '11 at 17:26
The `input` argument to `communicate` is a plain string, not a stream, so if the input data might be large, put it in a producer thread. If the output data might be large, read incrementally from the output as in [this example](http://stackoverflow.com/a/14026178/33208). – Jed Dec 24 '12 at 22:55

subprocess.Popen and buffered process output

2 Answers2