Python getstatusoutput replacement not returning full output

Question

I came across this great replacment for the getstatusoutput() function in Python 2.* which works equally well on Unix and Windows. However, I think there is something wrong with the way the output is constructed. It only returns the last line of the output, but I can't figure out why. Any help would be awesome.

def getstatusoutput(cmd):
    """Return (status, output) of executing cmd in a shell."""
    """This new implementation should work on all platforms."""
    import subprocess
    pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True, universal_newlines=True)
    output = "".join(pipe.stdout.readlines())
    sts = pipe.returncode
    if sts is None: sts = 0
    return sts, output

Why don't you use `subprocess.check_call` inside a try block? You can get the return code (if it's non-zero) in the `except` part. — wim, Apr 18 '12 at 02:50

score 2 · Accepted Answer · answered Apr 18 '12 at 03:09

2

There is a bug in this.

The call:

pipe = subprocess.Popen(...)

kicks off the process. The line:

output = "".join(pipe.stdout.readlines())

reads the process's stdout (if there were error output it could get jammed up, but since you did not redirect stderr this is safe; in the more general case, you need to use the .communicate() function to avoid possible wedging). You might think: well, now that I've read all the output, the subprocess has clearly finished, so pipe.returncode should be set. But in fact, if you replace:

sts = pipe.returncode
if sts is None: sts = 0

with code that includes a diagnostic of some sort (or just delete the second line entirely), you'll find that, at least on some systems sometimes, sts is None. The reason is that the subprocess module has not had a chance to retrieve it yet. You should replace those lines with:

sts = pipe.wait()

to collect the result and obviate the need for the is None test. (It's safe to call .wait() even if you've used .communicate() and/or already called .wait().)

The "".join(...) can be done slightly more efficiently by using just:

output = pipe.stdout.read()

All that said, it does get the full output for me. Perhaps you're reading from something that uses just '\r' for newlines and your Python was built without universal-newline support? (If I run something that produces \r-for-newline I still get all the lines, separated by actual newlines.)

answered Apr 18 '12 at 03:09

torek

448,244
59
642
775

thanks for this excellent explanation. I've learned a lot already. However, I think you're onto something with the newline thing. I am only getting `'\n'` back from my function. Can you please expand on the newline issue? – MFB Apr 18 '12 at 04:49
Per the [docs](http://docs.python.org/library/functions.html#open), "universal newline" means: when reading an input stream, an input line that "ends with" any of plain `\r` or `\r\n` or plain `\n` will make Python "see" just a simple `\n` character. But this requires that Python be built with the support. If not, a plain `\r` would be seen as `\r`, and if you `print` the resulting string you might not see everything. For example: `>>> print 'something\relse'` *appears* to print `elsething`. – torek Apr 18 '12 at 04:55
That makes sense, but shouldn't I be getting all the output with `\n` mixed in? Instead I am simply getting a single `'\n'` even when I know the subprocess (ffmpeg in this case) is outputting a bunch more. – MFB Apr 18 '12 at 05:00
You should get everything it sends to `stdout`. Maybe it sends other stuff to `stderr`? And: yes, it does: http://stackoverflow.com/questions/1455532/ffmpeg-and-pythons-subprocess – torek Apr 18 '12 at 05:44
Thanks torek. Redirecting stderr didn't work either, however one of the posts on your link suggests using `pexpect`. Its approximately one billion times easier to implement and outputs just fine (well, with a few nasty escape chars). Appreciate your time very much – MFB Apr 18 '12 at 23:49
Yes, `pexpect` is the way to go for more complex cases, including programs that insist on being "interactive" even when they are not. :-) – torek Apr 19 '12 at 00:09

Python getstatusoutput replacement not returning full output

1 Answers1

Linked