5

I'd like to run a system process, intercept the output, and modify it real-time, line by line, in a Python script.

My best attempt, which waits for the process to complete before printing, is:

#!/usr/bin/env python
import subprocess

cmd = "waitsome.py"
proc = subprocess.Popen(cmd, shell=True, bufsize=256, stdout=subprocess.PIPE)
for line in proc.stdout:
    print ">>> " + line.rstrip()

The script waitsome.py simply prints a line every half a second:

#!/usr/bin/env python
import time
from sys import stdout

print "Starting"
for i in range(0,20):
    time.sleep(0.5)
    print "Hello, iteration", i
    stdout.flush()

Is there an easy solution to get subprocess to allow iterating over the output in real time? Do I have to use threads?

Once upon a time, I scripted in Perl, and this was a piece of cake:

open(CMD, "waitsome.py |");
while (<CMD>) {
    print ">>> $_";
}
close(CMD);
Seth Johnson
  • 14,762
  • 6
  • 59
  • 85
  • Duplicate: http://stackoverflow.com/search?q=%5Bpython%5D+subprocess+real-time, specifically http://stackoverflow.com/questions/527197/intercepting-stdout-of-a-subprocess-while-it-is-running, http://stackoverflow.com/questions/803265/getting-realtime-output-using-subprocess, – S.Lott Jul 05 '09 at 23:34
  • Sorry, I only saw the first one and understood it to be a problem with the *subprocess'* buffering, not that of the parent python script. – Seth Johnson Jul 06 '09 at 02:13

2 Answers2

14

Looping over a file unavoidably buffers things in pretty large chunks -- a known issue with all Python 2.* implementations. It works as you intend in Python 3.1, with the final loop being slightly different:

for line in proc.stdout:
    print(">>> " + str(line.rstrip()))

If upgrading to Python 3.1 is impractical (and I know it will often be!), go the other way and write the loop in an old-fashioned manner -- the following version of the loop does work as you intend in Python 2.*:

while True:
    line = proc.stdout.readline()
    if not line:
        break
    print ">>> " + line.rstrip()
Alex Martelli
  • 854,459
  • 170
  • 1,222
  • 1,395
  • 1
    Yep, but now that Python 3.1 is out (with a much better implementation of the whole I/O stack) there's no reason to stick with 3.0 (it was definitely a transitional version;-). – Alex Martelli Jul 06 '09 at 02:16
  • 1
    `for line in iter(proc.stdout.readline, ''): print ">>>", line,` could be used instead of the while loop on Python 2. – jfs Dec 21 '13 at 05:56
0

This whole thing can be encapsulated in an iterator as:

def subprocess_readlines(out):
    while True:
        line = out.readline()
        if not line:
            return
        yield line

And called as:

for line in subprocess_readlines(proc.stdout):
    print ">>>", line.rstrip()
Jeff Younker
  • 179
  • 2
  • 5
  • Buffering in Python has changed a lot since I wrote the original question :) Yes, what took a lot of effort for python 2.5 can now be done in a few lines. – Seth Johnson Feb 25 '18 at 01:07
  • I dont understand where subprocess_readlines needs to be defined for this approach. Can you elaborate? – 0-0 Mar 08 '20 at 23:20
  • Ooops. I'd not added the call to `subprocess_readlines()`. I just updated that. – Jeff Younker Mar 10 '20 at 00:04