51

I want to redirect the stderr output of a subprocess to stdout. The constant STDOUT should do that, shouldn't it?

However,

$ python >/dev/null -c 'import subprocess;\
                        subprocess.call(["ls", "/404"],stderr=subprocess.STDOUT)'

does output something. Why is that the case, and how do I get the error message on stdout?

phihag
  • 278,196
  • 72
  • 453
  • 469

2 Answers2

51

In Python < v3.5:

A close read of the source code gives the answer. In particular, the documentation is misleading when it says:

subprocess.STDOUT
Special value that (...) indicates that standard error should go into the same handle as standard output.

Since stdout is set to "default" (-1, technically) when stderr=subprocess.STDOUT is evaluated, stderr is set to "default" as well. Unfortunately, this means that stderr output still goes to stderr.

To solve the problem, pass in the stdout file instead of subprocess.STDOUT:

$ python >/dev/null -c 'import subprocess,sys;subprocess.call(["ls", "/404"],
                        stderr=sys.stdout.buffer)'

Or, for compatibility with legacy 2.x versions of Python:

$ python >/dev/null -c 'import subprocess,sys;subprocess.call(["ls", "/404"],
                        stderr=sys.stdout.fileno())'
Macke
  • 24,812
  • 7
  • 82
  • 118
phihag
  • 278,196
  • 72
  • 453
  • 469
  • 2
    One can leave stderr as-is too: `stderr=sys.stderr.fileno()` . Nice example; I am going to simply use this technique in all my scripts. We have legacy Python 2.x; thus I'll know exactly where I told it to go without confusion. – Mike S Sep 01 '15 at 13:52
  • 2
    For the record this is no-longer necessary in python 3.5+, as this case is automatically handled: https://github.com/python/cpython/blob/75c1ca7b6c546ef674eb40bfe47f41e6045dc99b/Lib/subprocess.py#L1143-L1147 – ipetrik Oct 09 '20 at 20:40
33

Actually, using subprocess.STDOUT does exactly what is stated in the documentation: it redirects stderr to stdout so that e.g.

command = ["/bin/ls", "/tmp", "/notthere"]
process = subprocess.Popen(command, shell=False, bufsize=1, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = ""
while (True):
    # Read line from stdout, break if EOF reached, append line to output
    line = process.stdout.readline()
    line = line.decode()
    if (line == ""): break
    output += line

results in variable output containing the process' output from both stdout and stderr.

stderr=subprocess.STDOUT redirects all stderr output directly to stdout of the calling process, which is a major difference.

EDIT: Updated code for newer Python versions:

command = ["/bin/ls", "/tmp", "/notthere"]
process = subprocess.Popen(command, shell=False, text=True, bufsize=1, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = ""
while (True):
    # Read line from stdout, break if EOF reached, append line to output
    line = process.stdout.readline()
    if (line == ""): break
    output += line
Fonic
  • 2,625
  • 23
  • 20
  • 2
    -1 This case is irrelevant to the question which is about `stdout` being `None`. And `subprocess.STDOUT` does not redirect to the "calling console" (I think you mean the file handle 1 that the current process was given), as your own example shows - if you call Popen with `['ls', '/404']`, the [error message is not displayed](http://ideone.com/2JhcRT). – phihag May 13 '13 at 07:27
  • 1
    You are somewhat right. However, I don't think that this is irrelevant at all, since this is intended to clarify your post regarding the documentation being misleading - which isn't the case. I would have left a comment below your post, but a year ago I wasn't allowed to do so. – Fonic Jul 10 '14 at 11:06
  • 1
    `python -c 'import subprocess,sys;subprocess.call(["ls", "/404"], stderr=subprocess.STDOUT)' 2>/dev/null` shows that the stderr of `ls` goes to stderr (file descriptor 2) because you will not see any output (presuming you have no `/404` on your system). That Python documentation really whiffed on that topic, and it's especially annoying to us shell scripters who are used to manipulating stdout/stderr in myriads of ways, and have to come here to SO just to figure out how to do something which should be simple and non-confusing. – Mike S Sep 01 '15 at 13:31
  • 2
    It seems to me that the above here is the simpler solution and the intention of the OP. – Ray Salemi Sep 14 '17 at 15:36