Reading stderr of subprocess while it is executing

Question

I'd like to read what is written to stderr by a subprocess while it is executing.

However, when I use this script that I've written, stderr does not seem to have anything for me to read until the subprocess has exited.

#!/usr/bin/env python2

import sys
from subprocess import Popen, PIPE, STDOUT

if len(sys.argv) < 2:
    print "Please provide a command"
    sys.exit(1)

sub = Popen(sys.argv[1:], stdout=PIPE, stderr=STDOUT)

for i, line in enumerate(sub.stdout):
    sys.stdout.write("%d: %s" % (i, line))

edit:

Ok, I've gotten closer now. If I specify the number of bytes to read it overcomes the buffering.

#!/usr/bin/env python2

import sys 
from subprocess import Popen, PIPE, STDOUT

if len(sys.argv) < 2:
    print "Please provide a command"
    sys.exit(1)

sub = Popen(sys.argv[1:], stdout=PIPE, stderr=STDOUT)

i = 0 
while sub.poll() is None:
    line = sub.stdout.read(64)
    line.strip("\b")
    sys.stdout.write("%d: %s\n" % (i, line))
    i += 1

Snippet of output:

58:  86 q=21.0 size= 4541841kB time=00:00:22.08 bitrate=1685014.2kbi
frame=  567 fps= 86 q=22.0 size= 4543667kB time=00:00:2
frame=  621 fps= 87 q=20.0 sizs/s    
frame= 4545352kB time=00:00:26.11 bitrate=1425939.2kbits/s    
62: =  686 fps= 90 q=12.0 size= 4546970kB time=00:00:28.89 bitrate=1
frame=  758 fps= 93 q=25.0 size= 4548534kB t
frame=  794 fps= 92 bitrate=1168185.5kbits/s    
65:  q=27.0 size= 4550901kB time=00:00:33.40 bitrate=1115897.0kbits/
frame=  827 fps= 91 q=27.0 size= 4552324kB time=00:00:34.7
frame=  857 fps= 89 q=26.0 size=  
frame=  254kB time=00:00:36.12 bitrate=1032874.9kbits/s    
69: 892 fps= 88 q=25.0 size= 4556598kB time=00:00:37.36 bitrate=9988
frame=  948 fps= 89 q=19.0 size= 4558565kB time=
frame= 1006 fps= 90 q=19937320.4kbits/s    
72: .0 size= 4560139kB time=00:00:42.16 bitrate=885880.0kbits/s    
73: frame= 1060 fps= 91 q=19.0 size= 4561958kB time=00:00:44.49 bitr
frame= 1122 fps= 93 q=18.0 size= 4563460
frame= 1173 fps=0:47.08 bitrate=793898.4kbits/s

It looks like my problem now is that ffmpeg is using backspace characters or similar to mess with stdout. Not sure what's happening here.

What process are you using? It is most likely buffering it's output when connected to a pipe. — Martijn Pieters, Mar 10 '13 at 21:44
See [catching stdout in realtime from subprocess](http://stackoverflow.com/q/1606795) — Martijn Pieters, Mar 10 '13 at 21:44
The process is ffmpeg, I am trying to read the fps during conversion — OregonTrail, Mar 10 '13 at 22:06
Would [Turn off buffering in pipe](http://unix.stackexchange.com/q/25372) help? — Martijn Pieters, Mar 10 '13 at 22:13

spinus · Answer 1 · 2013-03-13T09:05:14.093

4

I suggest to use sh module. It is very nice piece of software which wraps subprocesses in python and gives you pythonic, beautiful interface which you will love. Look at the docs.

If you really do not want sh module, use communicate method of Popen

edited Mar 13 '13 at 09:05

answered Mar 10 '13 at 21:45

spinus

5,497
2
20
26

Hai Vu · Answer 2 · 2013-03-10T22:19:21.867

1

You should use PIPE for both, also, you need to call communicate():

sub = Popen(sys.argv[1:], stdout=PIPE, stderr=PIPE)
output, error_output = sub.communicate()

print 'OUTPUT:'
print output

print 'ERROR:'
print error_output

edited Mar 10 '13 at 22:19

answered Mar 10 '13 at 21:58

Hai Vu

37,849
11
66
93

Why is that? Redirecting `stderr` to stdout is fine, and not the issue. – Martijn Pieters Mar 10 '13 at 22:07
In this case, it is not an issue. In general, you might want to separate them to know if there is any error occurred. I failed to mention the missing `communicate()`, which is essential to fix the original poster's problem. – Hai Vu Mar 10 '13 at 22:18

score 1 · Answer 3 · edited May 23 '17 at 12:17

1

If the subprocess doesn't use block-buffering then you could try:

for i, line in enumerate(iter(sub.stdout.readline, b"")):
    print i, line,

edited May 23 '17 at 12:17

Community

1
1

answered Mar 10 '13 at 22:19

jfs

399,953
195
994
1,670

Reading stderr of subprocess while it is executing

3 Answers3