1

I've got a script parent.py trying to to read stdout from a subprocess sub.py in Python.

The parent parent.py:

#!/usr/bin/python
import subprocess
p = subprocess.Popen("sub.py", stdout=subprocess.PIPE)
print p.stdout.read(1)

And the subprocess, sub.py:

#!/usr/bin/python
print raw_input( "hello world!" )

I would expect running parent.py to print the 'h' from "hello world!". Actually, it hangs. I can only get my expected behaviour by adding -u to sub.py's she-bang line.

This confuses me because the -u switch makes no difference when sub.py is run directly from a shell; the shell is somehow privy to the un-flushed output stream, unlike parent.py.

My goal is to run a C program as the subprocess, so I won't be able to control whether or not it flushes stdout. How is it that a shell has better access to a process's stdout than Python running the same thing from subprocess.Popen? Am I going to be able to read such a stdout stream from a C program that doesn't flush its buffers?

EDIT:

Here is an updated example based on korylprince's comment...

## capitalize.sh ##
#!/bin/sh

while [ 1 ]; do
    read s
    echo $s | tr '[:lower:]' '[:upper:]'
done

########################################

## parent.py ##
#!/usr/bin/python 
from subprocess import Popen, PIPE

# cmd = [ 'capitalize.sh' ] # This would work
cmd = [ 'script', '-q', '-f', '-c', 'capitalize.sh', '/dev/null']

p = Popen(cmd, stdin=PIPE)
p.stdin.write("some string\n")
p.wait()

When running through script, I get steady printing of newlines (and if this were a Python, subprocess, it'd raise an EOFerror).

ajwood
  • 18,227
  • 15
  • 61
  • 104

2 Answers2

1

An alternative is

p = subprocess.Popen(["python", "-u", "sub.py"], stdout=subprocess.PIPE)

or the suggestions here.

My experience is that yes, you will be able to read from most C programs without any extra effort.

The Python interpreter takes extra steps to buffer its output which is why it needs the -u switch to disable output buffering. Your typical C program won't do this.

I haven't run into any program (C or otherwise) other than the Python interpreter that I expected to work and didn't within a subshell.

Community
  • 1
  • 1
jedwards
  • 29,432
  • 3
  • 65
  • 92
  • 1
    I dealt with this problem before. Not all C programs will work. There is an awesome unix command called script that will make things work for you though. See my question that uses it similarly: http://stackoverflow.com/questions/11131825/main-python-process-is-stopped-using-subprocess-calls-in-socketserver – korylprince Sep 28 '12 at 03:34
  • It not working with my `c_prog`. If I pipe it's stdout to a file, and then keyboard-interrupt, the file will be empty. But if I let it terminate naturally, the output makes it to the file.. I suspect this is related. – ajwood Sep 28 '12 at 12:09
  • @korylprince: When I run the subproces through `script` using PIPEs for both std and stdout, the subprocess seems to fail with an EOFError. – ajwood Sep 28 '12 at 16:13
  • @korylprince: Actually, even if the only pipe is for stdin, subprocess fails when run though `script` – ajwood Sep 28 '12 at 16:15
0

The reason the shell can read output immediately, regardless of "-u" is because the program you're launching from the shell has its output connected to a TTY. When the stdout is connected to a TTY, it is unbuffered (because it is up to the TTY to buffer). When you launch the python subprocess from within python, you're connecting stdout to a pipe, which means you're at the mercy of the subprocess to flush its output when it feels like it.

If you're looking to do complicated interactions with a subprocess, look into this tutorial.

amoffat
  • 696
  • 4
  • 12