24

I am trying to read from a process that produces long and time-consuming output. However, I want to catch it's output as and when it is produced. But using something like the following seems to be buffering the command's output, so I end up getting the output lines all at once:

p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, bufsize=0)
    for line in p.stdout:
        print line

I am trying this on MacOS 10.5

rypel
  • 4,686
  • 2
  • 25
  • 36
Abhi
  • 243
  • 2
  • 4
  • Duplicate: http://stackoverflow.com/questions/874815/how-do-i-get-real-time-information-back-from-a-subprocess-popen-in-python-2-5, http://stackoverflow.com/questions/527197/intercepting-stdout-of-a-subprocess-while-it-is-running – S.Lott Jul 26 '09 at 16:54

3 Answers3

29

The file iterator is doing some internal buffering on its own. Try this:

line = p.stdout.readline()
while line:
    print line
    line = p.stdout.readline()

You also need to make sure the process you are running is actually flushing its output buffers frequently.

Calmarius
  • 18,570
  • 18
  • 110
  • 157
brendan
  • 2,643
  • 1
  • 20
  • 13
  • Thank you, you saved me some hair pulling. – haridsv Feb 13 '14 at 10:55
  • You also need to make sure the process you are running is actually flushing its output buffers frequently. I debugged more than half day to find out that stdout has quite a big buffer when directed into a pipe. So I had to add some flushes into the program's code to make the output actually appear. – Calmarius Jan 17 '20 at 17:39
6

Usually, every program will do more buffering on its input and/or output channels than you appear to desire... unless it's fooled into believing said channel's actually a terminal!

For that "fooling in a good cause" purpose, use pexpect -- it works just fine on a Mac (life is harder on Windows, though there are solutions that might help even there - fortunately we don't need to dwell on those as you use a Mac instead).

Alex Martelli
  • 854,459
  • 170
  • 1,222
  • 1,395
3

This was actually a bug that's fixed in Python 2.6: http://bugs.python.org/issue3907

Ari
  • 2,311
  • 1
  • 17
  • 17
  • 4
    On python 2.7 `for line in p.stdout` still delays its output while `for line in iter(p.stdout.readline, b'')` works as expected (it produces lines as soon as they are available). – jfs Sep 27 '12 at 18:43
  • Sorry, 2.x won't use the new io library by default, as the last comment on the bug report says; You'd have to do something like `for line in io.open(p.stdin.fileno())`. – Ari Jan 30 '13 at 23:42
  • 1
    @Ari: Should that be `for line in io.open(p.stdout.fileno())`? – Richard Cook Apr 21 '15 at 18:47