subprocess.Popen handling stdout and stderr as they come

Question

I'm trying to process both stdout and stderr from a subprocess.Popen call that captures both via subprocess.PIPE but would like to handle the output (for example printing them on the terminal) as it comes.

All the current solutions that I've seen will wait for the completion of the Popen call to ensure that all of the stdout and stderr is captured so that then it can be processed.

This is an example Python script with mixed output that I can't seem to replicate the order when processing it in real time (or as real time as I can):

$ cat mix_out.py

import sys

sys.stdout.write('this is an stdout line\n')
sys.stdout.write('this is an stdout line\n')
sys.stderr.write('this is an stderr line\n')
sys.stderr.write('this is an stderr line\n')
sys.stderr.write('this is an stderr line\n')
sys.stdout.write('this is an stdout line\n')
sys.stderr.write('this is an stderr line\n')
sys.stdout.write('this is an stdout line\n')

The one approach that seems that it might work would be using threads, because then the reading would be asynchronous, and could be processed as subprocess is yielding the output.

The current implementation of this just process stdout first and stderr last, which can be deceiving if the output was originally alternating between both:

cmd = ['python', 'mix_out.py']

process = subprocess.Popen(
    cmd,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    close_fds=True,
    **kw
)

if process.stdout:
    while True:
        out = process.stdout.readline()
        if out == '' and process.poll() is not None:
            break
        if out != '':
            print 'stdout: %s' % out
            sys.stdout.flush()

if process.stderr:
    while True:
        err = process.stderr.readline()
        if err == '' and process.poll() is not None:
            break
        if err != '':
            print 'stderr: %s' % err
            sys.stderr.flush()

If I run the above (saved as out.py) to handle the mix_out.py example script from above, the streams are (as expected) handled in order:

$ python out.py
stdout: this is an stdout line
stdout: this is an stdout line
stdout: this is an stdout line
stdout: this is an stdout line
stderr: this is an stderr line
stderr: this is an stderr line
stderr: this is an stderr line
stderr: this is an stderr line

I understand that some system calls might buffer, and I am OK with that, the one thing I am looking to solve is respecting the order of the streams as they happened.

Is there a way to be able to process both stdout and stderr as it comes from subprocess without having to use threads? (the code gets executed in restricted remote systems where threading is not possible).

The need to differentiate stdout from stderr is a must (as shown in the example output)

Ideally, no extra libraries would be best (e.g. I know pexpect solves this)

A lot of examples out there mention the use of select but I have failed to come up with something that would preserve the order of the output with it.

I really can't think of any solution that doesn't use threads. Is there any particular reason you want to avoid them? — dano, Oct 14 '14 at 21:02
If all you did want to do is print to terminal, that's the default if you take out the `PIPE`s. Also it wouldn't guarantee exactly the right order, but could you put the `stdout` and `stderr` reads in the same `while True` loop? — GP89, Oct 15 '14 at 09:43
@dano I can't use threads because this code would get executed in remote (restricted) systems where threads cannot be spawned — alfredodeza, Oct 23 '14 at 14:47
related: [Subprocess.Popen: cloning stdout and stderr both to terminal and variables](http://stackoverflow.com/q/17190221/4279) — jfs, Oct 28 '14 at 17:07
related: [Displaying subprocess output to stdout and redirecting it](http://stackoverflow.com/q/25750468/4279) — jfs, Oct 28 '14 at 17:09

Brandt · Answer 1 · 2022-05-06T12:31:55.030

5

If you are looking for a way of having subprocess.Popen` output to stdout/stderr in realtime, you should be able to achieve that with:

import sys, subprocess
p = subprocess.Popen(cmdline,
                     stdout=sys.stdout,
                     stderr=sys.stderr)

Maybe using stderr=subprocess.STDOUT may simplify your filtering, IMO.

edited May 06 '22 at 12:31

answered Sep 11 '16 at 12:10

Brandt

5,058
3
28
46

score 0 · Answer 2 · answered Jan 16 '19 at 09:43

0

I found working example here (see listing of capture_together.py). Compiled C++ code that mixes cerr and cout executed as subprocess on both Windows and UNIX OSes. Results are identitical

answered Jan 16 '19 at 09:43

dyomas

700
5
13

score -1 · Accepted Answer · edited May 23 '17 at 12:08

-1

I was able to solve this by using select.select()

process = subprocess.Popen(
    cmd,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    close_fds=True,
    **kw
)

while True:
    reads, _, _ = select(
        [process.stdout.fileno(), process.stderr.fileno()],
        [], []
    )

    for descriptor in reads:
        if descriptor == process.stdout.fileno():
            read = process.stdout.readline()
            if read:
                print 'stdout: %s' % read

        if descriptor == process.stderr.fileno():
            read = process.stderr.readline()
            if read:
                print 'stderr: %s' % read
        sys.stdout.flush()

    if process.poll() is not None:
        break

By passing in the file descriptors to select() on the reads argument (first argument for select()) and looping over them (as long as process.poll()indicated that the process was still alive).

No need for threads. Code was adapted from this stackoverflow answer

edited May 23 '17 at 12:08

Community

1
1

answered Oct 28 '14 at 14:40

alfredodeza

5,058
4
35
44

Yes, I've tested it: it does not preserve the order. To disable buffering, use `-u` flag (if the child is `python`) or [use `stdbuf` utility (or its analogs) or pseudo-tty (`pty`, `pexpect` modules)](http://stackoverflow.com/q/20503671/4279). Some programs provide a special flag e.g., `grep`'s `--line-buffered`. – jfs Oct 28 '14 at 17:20
you cannot control buffering of other tools/shells that buffer. The code in this answer does work and preserves the order. – alfredodeza Oct 28 '14 at 18:27
you can control the buffering -- my previous comment mentions several methods. Try your code with `mix_out.py` from your question. Your code won't preserve the order unless you pass `-u` flag to `python` executable *or* run it using `stdbuf` or run it with pseudo-tty. – jfs Oct 28 '14 at 18:33

subprocess.Popen handling stdout and stderr as they come

3 Answers3