How to read output from subprocess Popen correctly?

Question

I'm trying to redirect stderr to stdout and read the output of a command. It looks like the code works, but when it reaches the end of the output, it throws an exception

Code:

with Popen(["bash", "-c", "for ((i=0;i<10;i++)); do echo $i; sleep .5; done"],
           stderr=subprocess.STDOUT) as proc:
    out, err = proc.communicate()
    for line in out:
        if line:
            print(line)

Output:

0
1
2
3
4
5
6
7
8
9
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-28-3c150d6a4092> in <module>()
      2            stderr=subprocess.STDOUT) as proc:
      3         out, err = proc.communicate()
----> 4         for line in out:
      5                 if line:
      6                         print(line)

TypeError: 'NoneType' object is not iterable

k107 · Answer 1 · 2015-04-12T15:21:53.010

1

Oh, looks like I need to set stdout=PIPE, and I don't need communicate()

with Popen(["bash", "-c", "for ((i=0;i<10;i++)); do echo $i; sleep .5; done"],
           stdout=subprocess.PIPE, stderr=subprocess.STDOUT) as proc:
    for line in proc.stdout:
        print(line.decode().rstrip())

With J.F. Sebastian's universal_newlines tip:

with Popen(["bash", "-c", "for ((i=0;i<10;i++)); do echo $i; sleep .5; done"],
           stdout=subprocess.PIPE, 
           stderr=subprocess.STDOUT, 
           universal_newlines=True) as proc:
    for line in proc.stdout:
        print(line, end='')

edited Apr 12 '15 at 15:21

answered Apr 09 '15 at 19:56

k107

15,882
11
61
59

you can drop `if line` condition; `line` is never empty here (a newline is not stripped). You could use `universal_newlines=True` to enable text mode (to avoid calling `line.decode()` manually). You could use just `print(line, end='')` in this case. – jfs Apr 09 '15 at 22:43

score 1 · Answer 2 · answered Apr 09 '15 at 20:35

I see, you can achive what you want using another approach, but I would like to explain what was happining with your first version.

The thing is you don't have any PIPE for output in proc, so the output don't "travel" through your script, it goes directly to the screen, thats the reason you see the output:

and then in the line:

out, err = proc.communicate() # out is None, since you don't 
                              # pass the argument "stdout=subprocess.PIPE"
                              # when creating proc.

and later when you try to iterate over out:

for line in out:  # TypeError: 'NoneType' object is not iterable
     ...          # We are talking about out variable here, which is None.
     print(line)  # This line never get hit.

Solution.

Here you have a little example redirecting stdout to stderr and using communicate:

import subprocess
import sys

proc = subprocess.Popen(['ls'], stderr=sys.stdout, shell=True) # Not the use of sys.stdout
out, err = proc.communicate()
print(err.decode("utf-8"))

score 0 · Answer 3 · answered Apr 09 '15 at 22:42

TypeError: 'NoneType' object is not iterable

You should add stdout=PIPE otherwise .communicate() always returns None for out.

btw, If your code uses stdout=PIPE, stderr=STDOUT then err is always None and out contains merged stdout/stderr as a single bytestring i.e., for byte in out yields one byte at a time, not line.

You could get stdout/stderr separately if you set stdout=PIPE, stderr=PIPE and call .communicate().

How to read output from subprocess Popen correctly?

3 Answers3

Solution.

Linked