Distinct, Merged, stdout and stderr from subprocess.Popen()

Question

Given an arbitrary executable that can write to either stdout or stderr arbitrarily quickly e.g.

#include <stdio.h>

int main(void)
{
    fprintf(stdout, "OUT:1\n");
    fprintf(stdout, "OUT:2\n");
    fprintf(stderr, "ERR:3\n");
    fprintf(stderr, "ERR:4\n");
    fprintf(stdout, "OUT:5\n");
    fprintf(stderr, "ERR:6\n");
    fprintf(stdout, "OUT:7\n");

    return 0;
}

Is it possible to get BOTH stdout and stderr merged ie.

stdout_master_fd, stdout_slave_fd = pty.openpty()
subprocess.Popen(stdout=stdout_slave_fd, stderr=subprocess.STDOUT)

And stdout and stderr separated.

stdout_master_fd, stdout_slave_fd = pty.openpty()
stderr_master_fd, stderr_slave_fd = pty.openpty()
subprocess.Popen(stdout=stdout_slave_fd, stderr=stderr_slave_fd)

With only one call to subprocess.Popen()

There are a number of similar questions:

The first link uses timestamps to try to sort stderr into stdout, but even when using ptys buffering seems to occur:

stdout - 2019-06-27T23:43:55.337389 - OUT:1
stdout - 2019-06-27T23:43:55.337389 - OUT:2
stdout - 2019-06-27T23:43:55.337389 - OUT:5
stdout - 2019-06-27T23:43:55.337389 - OUT:7
stderr - 2019-06-27T23:43:55.337413 - ERR:3
stderr - 2019-06-27T23:43:55.337413 - ERR:4
stderr - 2019-06-27T23:43:55.337413 - ERR:6

Plus this obviously can't handle arbitrarily fast writes.

The second link uses poll() but only reads about two lines, and the exact number changes, and isn't ordered correctly:

STDOUT:root:OUT:1
STDERR:root:ERR:3
STDOUT:root:OUT:2

I think the second method has a better premise behind it, but neither are currently looking too promising.

My version of the second method, modified from jfs in the comments of the second link.

def execute(cmd):
        import subprocess, pty, os, select, logging

        logging.basicConfig(level=logging.INFO)
        logging.addLevelName(logging.INFO+1, 'STDOUT')
        logging.addLevelName(logging.INFO+2, 'STDERR')
        logger = logging.getLogger()

        stdout_master_fd, stdout_slave_fd = pty.openpty()
        stderr_master_fd, stderr_slave_fd = pty.openpty()

        p = subprocess.Popen(cmd, stdout=stdout_slave_fd, stderr=stderr_slave_fd, close_fds=True)

        os.close(stdout_slave_fd)
        os.close(stderr_slave_fd)

        with os.fdopen(stdout_master_fd)as stdout, os.fdopen(stderr_master_fd) as stderr:
            poll = select.poll()
            poll.register(stdout, select.POLLIN)
            poll.register(stderr, select.POLLIN | select.EPOLLHUP)

            def cleanup(_done=[]):
                if _done:
                    return
                _done.append(1)
                poll.unregister(stderr)
                poll.unregister(stdout)
                assert p.poll() is not None

            read_write = {stdout.fileno(): (stdout.readline, lambda s: logger.log(logging.INFO+1, s)),
                          stderr.fileno(): (stderr.readline, lambda s: logger.log(logging.INFO+2, s))}

            while True:
                events = poll.poll(40)

                if not events and p.poll() is not None:
                    cleanup()
                    break

                for fd, event in events:
                    if event & select.POLLIN:
                        read, write = read_write[fd]
                        line = read()
                        if line:
                            write(line.rstrip())
                    elif event & select.POLLHUP:
                        cleanup()
                    else:
                        assert 0

        p.wait()
        return p

execute(['./test'])

The overall outcome would preferably be something along the lines of:

outs['stdout'] == """
OUT:1
OUT:2
OUT:5
OUT:7
"""

outs['stderr'] == """
ERR:3
ERR:4
ERR:6
"""

outs['merged'] == """
OUT:1
OUT:2
ERR:3
ERR:4
OUT:5
ERR:6
OUT:7
"""

And direct access to the return value of subprocess.Popen(), even if a library was involved.

Distinct, Merged, stdout and stderr from subprocess.Popen()

0 Answers0