Real time output of subprocess.popen() and not line by line

Question

I'm currently rewriting a little wrapper program in python that I once wrote in C++. It extracts files from a file and boxes them in another format.

In C++ the output from the system commands I need to run was "real time" i.e the status bar and the percentage indicator of some commands where shown in real time. With python I get each 'percent' dumped on the screen individually (because I read it line by line). Here's an example: Thats how a status bar looks in the python version (this goes on until 100). In C++ it does update itself though.

| (02/100)\rImporting AVC-H264: |                    | (03/100)\rImporting AVC-H264: |                       | (04/100)\rImporting AVC-H264: |=

Thats the corresponding python code:

p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for line in iter(p.stdout.readline, ""):
   print line,

Any ideas on how I could make it look like in C++ ?

score 7 · Accepted Answer · edited Feb 22 '11 at 14:58

7

Could be two things...

It's likely that readline is changing some things from the output of your program. I believe \r is carriage return and tells the terminal to return to the beginning of the line and then the program can output overtop the text it just output. Readline is most likely removing this.

First thing to try,

p = subprocess.Popen(args, stdout=subprocess.PIPE, \
                     stderr=subprocess.PIPE, \
                     universal_newlines=True)
for line in iter(p.stdout.readline, ""):
    sys.stdout.write('\r'+line[:-1])
    sys.stdout.flush()

You have to do the flush because stdout buffers until it gets a \n and of course you're not writing one.

edited Feb 22 '11 at 14:58

Macke

24,812
7
82
118

answered Jan 07 '11 at 05:18

milkypostman

2,955
27
23

Thanks for all your answers. I play around with this solution, and I think it has to be something along the lines of this: while 1: data = p.stdout.read() sys.stdout.write(data) # process is done so break if p.returncode is not None: break The problem is that now it dumps everything on the screen at once, but underneath each other and it still doesn't override itself. I have trouble understanding what p.stdout.read() returns. At which points does it decide "Thats what I'm going to return"? – Basil Jan 07 '11 at 21:18
p.stdout.read() will read from stdout any and all data that is available on that pipe. I've edited my solution. Give that a try and let me know what doesn't work. – milkypostman Jan 07 '11 at 23:03
Thanks for your help! It works better now, the lines don't show up underneath each other any more. What happens now is that I see the first two lines of the command output and the space underneath (where 'Progress: (percentage) ' should be is empty. As soon as the command finishes, 'Progress:100%' is printed there. I think it has something to do with the buffer. It makes no difference if I omit 'sys.stdout.flush()'. Please let me know if you need more detailed info. – Basil Jan 08 '11 at 18:27
Hey, I posted a solution that works for the demo I built. Basically, it seems like neither `readline` nor `read` were processing the lines because it didn't recognize `\r` as a newline character. It says output is unbuffered but there *has* be some buffering going on that I can say I don't completely understand (and it's likely different on different systems). Give this a shot and let me know. – milkypostman Jan 08 '11 at 19:28
@Basil: drop `stderr=PIPE`. Either set `stderr=STDOUT` or [`stderr=DEVNULL`](http://stackoverflow.com/a/11270665/4279). Do not use `PIPE` unless you read from the stream. Otherwise the child process may hang if the corresponding OS pipe buffer fills up. Also, you should use `line.rstrip('\n')` instead of `line[:-1]` in case there is no newline at the end of the output. Even if you use `bufsize=1`, the buffering issues might not end, see [Python C program subprocess hangs at “for line in iter”](http://stackoverflow.com/q/20503671/4279) – jfs Sep 10 '14 at 05:07

score 1 · Answer 2 · edited May 23 '17 at 12:26

1

On Windows, you can output the backspace character (ASCII code 8). Example (prints the current iteration using only single digit numbers):

>>> import time
>>> import sys
>>> for i in xrange(10):
...     sys.stdout.write(str(i))
...     time.sleep(.5)
...     sys.stdout.write(chr(8))
...

You would need to keep track of the number of characters on the current line... I'm sure there must be a better way.

On Linux, it appears that you can write carriage returns to reset the cursor position. See Erase the current printed console line and In-place progress output in the terminal or console.

edited May 23 '17 at 12:26

Community

1
1

answered Jan 06 '11 at 22:40

Jeremy Brown

17,880
4
35
28

I think you can get away without knowing the number of characters because the output from the parent process is always the same length as far as I can tell. – milkypostman Jan 07 '11 at 23:07

score 0 · Answer 3 · answered Jan 06 '11 at 22:31

0

Sorry but i didn't understand well the real time part, but maybe i can help with "update it self" part, try this:

for line in iter(p.stdout.readline, ""):
   print line + '\r',

answered Jan 06 '11 at 22:31

mouad

67,571
18
114
106

Real time output of subprocess.popen() and not line by line

3 Answers3

Linked