python subprocess proc.stderr.read() introduce extra lines?

Question

I want to run some command and grab whatever is output to stderr. I have two versions of function that does this version 1.

def Getstatusoutput(cmd):
    """Return (status, output) of executing cmd in a shell."""

    import sys
    mswindows = (sys.platform == "win32")

    import os
    if not mswindows:
        cmd = '{ ' + cmd + '; }'

    pipe = os.popen(cmd + ' 2>&1', 'r')
    text = pipe.read()
    sts = pipe.close()
    if sts is None: sts = 0
    if text[-1:] == '\n': text = text[:-1]
    return sts, text

and version 2

def Getstatusoutput2(cmd):
    proc = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
    return_code = proc.wait()
    return return_code, proc.stdout.read(), proc.stderr.read()

The first version prints stderr output as I expect. The second version prints one blank line after every line. I suspect this is due to text[-1:] line in the version 1...but I can't seem to do something similar in second version. Can anybody explain what I need to do to make second function generate the same output as first one without extra lines in between (and at the very end) ?

Update: Here's how I am printing the output Here's how I am printing

      status, output, error = Getstatusoutput2(cmd)
      s, oldOutput = Getstatusoutput(cmd)
      print "oldOutput = <<%s>>" % (oldOutput)
      print "error = <<%s>>" % (error)

How are you printing? The `print` function adds newlines too if you don't call it correctly... — Chris Eberle, Nov 02 '11 at 18:27
None of that code is printing anything. Where is your code that prints out the output? — Adam Rosenfield, Nov 02 '11 at 18:27
status, output, error = Getstatusoutput2(cmd) s, oldOutput = Getstatusoutput(cmd) print "oldOutput = <<%s>>" % (oldOutput) print "error = <<%s>>" % (error) — user536012, Nov 02 '11 at 18:31
Are you sure that your command isn't printing two newlines between every line instead of one? — Adam Rosenfield, Nov 02 '11 at 18:52

score 2 · Answer 1 · edited Jun 20 '20 at 09:12

You could use subprocess.check_output([cmd], stderr=STDOUT) to capture all output.

To capture stdout, stderr separately you could use .communicate():

stdout, stderr = Popen([cmd], stdout=PIPE, stderr=PIPE).communicate()

To get all lines without a newline character at the end you could call stderr.splitlines().

To avoid printing additional newline if it is already present add ',' after the variable in a print statement:

print line,

Or if you use print() function:

print(line, end='')

Note

Your Getstatusoutput2() will block if the cmd produces enough output, use above solutions instead:

>>> len(Getstatusoutput2(['python', '-c',"""print "*"*2**6"""])[1])
65
>>> len(Getstatusoutput2(['python', '-c',"""print "*"*2**16"""])[1])

Popen.wait() documentation:

Wait for child process to terminate. Set and return returncode attribute.

Warning: This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.

Related Use communicate() rather than stdin.write(), stdout.read() or stderr.read()

+1 subprocess.check_output() is the easiest way to get the code right when capturing data from a subprocess. — Raymond Hettinger, Nov 02 '11 at 18:35

score 2 · Answer 2 · answered Nov 02 '11 at 18:31

You can add .strip():

def Getstatusoutput2(cmd):
    proc = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
    return_code = proc.wait()
    return return_code, proc.stdout.read().strip(), proc.stderr.read().strip()

Python string Docs:

string.strip(s[, chars])

Return a copy of the string with leading and trailing characters removed. If chars is omitted or None, whitespace characters are removed. If given and not None, chars must be a string; the characters in the string will be stripped from the both ends of the string this method is called on.

string.whitespace

A string containing all characters that are considered whitespace. On most systems this includes the characters space, tab, linefeed, return, formfeed, and vertical tab.

python subprocess proc.stderr.read() introduce extra lines?

2 Answers2

Note

Linked