Getting progress message from a subprocess

Question

I want to start a program which needs several minutes to complete. During this time I want to read the progress message of the program (which are printed on the stdout). The problem is that I cannot find a way to read out its output during its run.

The only function I found to read out the output of a program is Popen.communicate(), but this method waits until the process finishes. So it is impossible to get the progress and make it visible to the user in a special formatted way.

Is it possible to do this another way?

When I run the process with subprocess.popen with my script I see the output of the program on the screen. Is it possible to hide it? (Ubuntu 10.10, normal terminal)

chmullig · Accepted Answer · 2011-02-09T22:32:05.657

23

Simplest is to call Popen with the keyword argument stdout=subprocess.PIPE.

p = subprocess.Popen(["ls"], stdout=subprocess.PIPE)
while True:
    line = p.stdout.readline()
    if not line:
        break
    print line

To see this in action, here are two sample scripts. Make them both in the same directory and run python superprint.py

printandwait.py:

import time
import sys
print 10
sys.stdout.flush()
time.sleep(10)
print 20
sys.stdout.flush()

superprint.py:

import subprocess
import sys
p = subprocess.Popen(["python printandwait.py"], shell=True, stdout=subprocess.PIPE)
while True:
    print "Looping"
    line = p.stdout.readline()
    if not line:
        break
    print line.strip()
    sys.stdout.flush()

edited Feb 09 '11 at 22:32

answered Feb 09 '11 at 22:16

chmullig

13,006
5
35
52

What if some data arrives after you've broken the while loop? – Neo Feb 09 '11 at 22:18
1

@Neo: his example breaks at EOF, which means no more data can arrive after that. BUT, to answer the implication of your question: if you don't keep reading the stdout pipe, the buffer may fill and the subprocess will block attempting to write stdout. – payne Feb 09 '11 at 22:26
It would be waiting for you in p.stdout to use later. However the readline call will block until new input arrives or the pipe closes. Added some example scripts. – chmullig Feb 09 '11 at 22:30
1

you could use `for line in iter(p.stdout.readline, b''): print line,` instead of the `while`-loop. – jfs Mar 10 '14 at 03:39

Neo · Answer 2 · 2011-02-09T22:31:40.843

3

You can do a poll on the status of your subprocess and keep outputting lines.

p = subprocess.Popen('ls;sleep 10', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

rc = p.poll()
while rc != 0:
    while True:
        line = p.stdout.readline()
        if not line:
            break
        print line
    rc = p.poll()

assert rc == 0

edited Feb 09 '11 at 22:31

answered Feb 09 '11 at 22:14

Neo

13,179
18
55
80

Vinay Sajip · Answer 3 · 2011-02-09T22:34:08.453

It's certainly possible: my package python-gnupg does exactly this, spawning gpg (Gnu Privacy Guard) under a subprocess. In the general case you need to specify subprocess.PIPE for the subprocess stdout and stderr; then create two separate threads which read the subprocess stdout and stderr to wherever you like.

In the case of python-gnupg, status messages from gpg are read and acted upon while the gpg process is running (not waiting until it's finished).

Basically, pseudocode is

process = subprocess.Popen(..., stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stderr = process.stderr
rr = threading.Thread(target=response_reader_func, args=(process.stderr,))
rr.setDaemon(True)
rr.start()

dr = threading.Thread(target=data_reader_func, args=(process.stdout,))
dr.setDaemon(True)
dr.start()

dr.join()
rr.join()
process.wait()

The reader functions are typically methods of an enclosing class which do the right thing based on what they're reading (in your case, updating progress info in some way).

Getting progress message from a subprocess

3 Answers3

Linked