Python subprocess capture stdout race condition

Question

When starting a process with POpen I want to capture the stderr and stdout pipes. My code works but has a race condition between the process being executed and the stream being wrapped. This manifests itself by some stdout bleed into the console before being wrapped when the scheduler returns to the Python program.

# Start process.
proc = subprocess.Popen(cmd_args, env=context, cwd=cwd,
    stdout=subprocess.PIPE if capture_out else subprocess.DEVNULL,
    stderr=subprocess.PIPE if capture_err else subprocess.DEVNULL,
    stdin=None)

# Wrap streams.
out_stream = NonBlockingStreamReader(proc.stdout) if capture_out else None
err_stream = NonBlockingStreamReader(proc.stderr) if capture_err else None

How can I wrap the streams before passing them into subprocess.Popen?

score 1 · Answer 1 · edited May 23 '17 at 12:24

This manifests itself by some stdout bleed into the console before being wrapped when the scheduler returns to the Python program.

There is no race.

stdout is either PIPE or DEVNULL in your case. stdout is redirected even before cmd_args is executed (after the fork but before exec() is called).

If you see any output at all; it means that cmd_args writes outside of stdout, stderr i.e., it may write to the terminal directly. See the first reason in Q: Why not just use a pipe (popen())?

You could use pty (pexpect), to capture such output, code example. There could be some issues if you want to provide multiple pseudo-ttys (to capture stdout/stderr separately).

Python subprocess capture stdout race condition

1 Answers1