Processing messages from a child process thorough stderr and stdout with Python

Question

My python code spawns the child process, and it prints out messages both stdout and stderr. I need to print them differently.

I have the following code to spawn child process and get the stdout result from it.

cmd = ["vsmake.exe", "-f"]
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, ''):
    print line,
    sys.stdout.flush()
    pass
p.wait()

How can I modify the code to check if the child process prints out message through stderr also?

ADDED

I need to print out the stderr and stdout as soon as the child process prints out something. And it is cross platform implementation, so it should run on Mac/Linux/PC.

Would you like to print both the messages on stdout and stderr in realtime? If yes, you will probably need threads on Windows (because there is neither `select()` nor non-blocking I/O). — Sven Marnach, Jan 26 '11 at 14:24
I edited my answer in order print both stderr and stdout as soon a line is printed in any of those two pipes. — Manuel Salvadores, Jan 26 '11 at 14:54

Manuel Salvadores · Accepted Answer · 2011-01-26T17:45:22.530

7

p = Popen(cmd, bufsize=1024,
stdin=PIPE, stdout=PIPE, stderr=PIPE, close_fds=True)
p.stdin.close()
print p.stdout.read() #This will print the standard output from the spawned process
print p.stderr.read() #This is what you need, error output <-----

So basically the error output gets redirected to the stderr Pipe.

If you need something more in real in time. I mean lines printed as soon as the spawned process prints something to stdout orstderr` then you can do something like:

def print_pipe(type_pipe,pipe):
    for line in iter(pipe.readline, ''):
         print "[%s] %s"%(type_pipe,line),

p = Popen(cmd, bufsize=1024,
stdin=PIPE, stdout=PIPE, stderr=PIPE, close_fds=True)

t1 = Thread(target=print_pipe, args=("stdout",p.stdout,))
t1.start()
t2 = Thread(target=print_pipe, args=("stderr",p.stderr,))
t2.start()

#optionally you can join the threads to wait till p is done. This is avoidable but it 
# really depends on the application.
t1.join()
t2.join()

In this case two threads will print every time that a line is written either to stdout or stderr. The parameter type_pipe just makes the distinction when the lines are printed to know if they are coming from stderr or stdout.

edited Jan 26 '11 at 17:45

answered Jan 26 '11 at 14:26

Manuel Salvadores

16,287
5
37
56

3

My favorite (for small outputs) is: `stdin,stdout = p.communicate()`. Gets it all done in one shot! Then do with stdin,stdout what you will. – MikeyB Jan 26 '11 at 14:31
nice one !! (+1) I did not know that one. – Manuel Salvadores Jan 26 '11 at 14:33
Given the example code in the question, I suspect the OP wants to print the output in real time. But we won't know this for sure until (s)he tells us. – Sven Marnach Jan 26 '11 at 14:39
@Sven : I added more to my question. – prosseek Jan 26 '11 at 14:48
1

You should fix your `print` statement -- the `%` operator will fail the way it is now. Also remember that `line` ends with "\n", and you don't want to print this twice. – Sven Marnach Jan 26 '11 at 15:01
Cheers @Sven, fixed ... errors of fast typing ;) +1 for the suggestions. – Manuel Salvadores Jan 26 '11 at 15:04
1

1. The first example can block if the output is sufficiently large. http://stackoverflow.com/questions/163542/python-how-do-i-pass-a-string-into-subprocess-popen-using-the-stdin-argument/165662#165662 2. Don't use `shell=True` unless you must. The semantic with a list `cmd` is different. 3. You could just add `,` at the end of `print` statement instead of `line.replace()`. 3. there should be `p.wait()` and `t1.join()`, `t2.join()` somewhere. – jfs Jan 26 '11 at 17:36
@J.F Sebastian the first example is just to introduce that stderr exists. About the other suggestions ... thanks for the `,` thing in the print I did not know I could do that. Very nice. About `p.wait()` or the `joins` ... it is discussable if you really need them. It depends on the application. You might not want to wait till `p` is finished. Anyway I have added a comment on it in the code. Thanks a lot for your suggestions. Cheers. – Manuel Salvadores Jan 26 '11 at 17:50

Sven Marnach · Answer 2 · 2011-01-26T23:27:06.543

1

The easiest way to do this platform-independently is using threads (unfortunately). Here is some example code:

def redirect_to_stdout(stream):
    for line in stream:
        sys.stdout.write(line)
        sys.stdout.flush()

cmd = ["vsmake.exe", "-f"]
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stderr_thread = threading.Thread(target=redirect_to_stdout, args=(p.stderr,))
stderr_thread.start()
redirect_to_stdout(p.stdout)
p.wait()
stderr_thread.join()

edited Jan 26 '11 at 23:27

answered Jan 26 '11 at 14:58

Sven Marnach

574,206
118
941
841

-1: s/run/start/. Use `stderr=subprocess.STDOUT` to redirect `stdout`. – jfs Jan 26 '11 at 17:29
@J.F. Sebastian: Thanks for pointing out the `start()`/`run()` confusion, corrected now. With `stderr=subprocess.STDOUT` it is impossible to meet the OP's requirement to "print [stdout and stderr] differently". If you could go that way, you would not need threads at all. – Sven Marnach Jan 26 '11 at 23:33
1

Exactly, your solution does not print stdout, stderr differently; it doesn't need threads in this case. – jfs Jan 27 '11 at 04:53

Processing messages from a child process thorough stderr and stdout with Python

ADDED

2 Answers2