13

I'm having problems redirecting stdio of another program using subprocess module. Just reading from stdout results in hanging, and Popen.communicate() works but it closes pipes after reading/writing. What's the easiest way to implement this?

I was playing around with this on windows:

import subprocess
proc = subprocess.Popen('python -c "while True: print \'Hi %s!\' % raw_input()"',
                        shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
                        stderr=subprocess.STDOUT)
while True:
    proc.stdin.write('world\n')
    proc_read = proc.stdout.readline()
    if proc_read:
        print proc_read
rypel
  • 4,686
  • 2
  • 25
  • 36
Ivan Baldin
  • 3,391
  • 3
  • 22
  • 14
  • Duplicate: http://stackoverflow.com/questions/163542/python-how-do-i-pass-a-string-into-subprocess-popen-using-the-stdin-argument, http://stackoverflow.com/questions/295459/how-do-i-use-subprocess-popen-to-connect-multiple-processes-by-pipes – S.Lott Jan 14 '09 at 14:24
  • `-u` flag would solve it for a Python subprocess. There are also [`pexpect`, `pty`](http://stackoverflow.com/a/12471855/4279) modules and [`unbuffer`, `stdbuf`, `script`](http://unix.stackexchange.com/q/25372/1321) utilities that can help to fix the block-buffering issue. – jfs May 22 '13 at 19:33

1 Answers1

21

Doesn't fit 100% to your example but helps to understand the underlying issue: Process P starts child C. Child C writes something to its stdout. stdout of C is a pipe which has a 4096 character buffer and the output is shorter than that. Now, C waits for some input. For C, everything is fine.

P waits for the output which will never come because the OS sees no reason to flush the output buffer of C (with so little data in it). Since P never gets the output of C, it will never write anything to C, so C hangs waiting for the input from P.

Fix: Use flush after every write to a pipe forcing the OS to send the data now.

In your case, adding proc.stdin.flush() in the main while loop and a sys.stdout.flush() in the child loop after the print should fix your problem.

You should also consider moving the code which reads from the other process into a thread. The idea here is that you can never know when the data will arrive and using a thread helps you to understand these issues while you write the code which processes the results.

At this place, I wanted to show you the new Python 2.6 documentation but it doesn't explain the flush issue, either :( Oh well ...

Aaron Digulla
  • 321,842
  • 108
  • 597
  • 820
  • 1
    Good explanation. Here's more from `pexpect` docs: [Q: Why not just use a pipe (popen())?](http://www.noah.org/wiki/pexpect#Q:_Why_not_just_use_a_pipe_.28popen.28.29.29.3F) – jfs May 22 '13 at 19:27
  • 2
    Pexpect docs has moved: here's [new link](http://pexpect.readthedocs.io/en/stable/FAQ.html#whynotpipe) – jfs Sep 13 '16 at 15:12