How can I call a command using Python, capturing stderr and stdout, without waiting for stderr/stdout to be closed?

Question

Right now I have some code that uses Popen.communicate() from subprocess (setting stdin=PIPE and stderr=PIPE) to run a command and capture both stderr and stdout.

The problem is that communicate() not only waits for the command to exit, it waits for stdout and stderr to be closed. The command I'm running spawns a child process which keeps stderr open, so even though the command is finished running (and shows as "defunct" in ps) communicate() is still hung.

I want to only wait for the command to finish without waiting on stderr/stdout. But I still want to capture any stderr/stdout output given while the command was running. The documentation for wait() is accompanied by a red box with a disclaimer:

This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.

Obviously, I also want to avoid deadlocks.

What is the right way to accomplish this task?

Maybe you'll get interesting information from http://stackoverflow.com/questions/803265/getting-realtime-output-using-subprocess or http://stackoverflow.com/questions/3140189/subprocess-popen-stdout-reading-stdout-in-real-time-again or http://stackoverflow.com/questions/17411966/printing-stdout-in-realtime-from-a-subprocess-that-requires-stdin — damienfrancois, Jan 15 '14 at 17:36
Could spawn separate threads to read the process's `stdout` and `stderr` so that you can call `wait()` without worrying about buffers filling up. Complicated part is if you want to stop reading `stderr`. See http://stackoverflow.com/questions/3076542/how-can-i-read-all-availably-data-from-subprocess-popen-stdout-non-blocking/ — Dan Getz, Jan 16 '14 at 23:51
@DanGetz thanks, I think that will work. Will report back if I get a working solution. — Caleb Spare, Jan 18 '14 at 01:05

score 1 · Answer 1 · answered Mar 04 '14 at 15:44

"shows as "defunct" in ps" implies that you might be on a system where select or fcntl would work i.e., you can read stdout/stderr without blocking easily.

Example: A starts B (cmd, the child), B starts C (grandchild), A reads output until B exits or EOF:

#!/usr/bin/env python
import os
from select import select
from subprocess import Popen, PIPE

p = Popen(cmd, stdout=PIPE, stderr=PIPE, bufsize=0)
read_set = [p.stdout, p.stderr]
pipename = {p.stdout: "stdout", p.stderr: "stderr"}
timeout = 0.5 # ugly but it works
while read_set and p.poll() is None: # while subprocess is running or until EOF
    for pipe in select(read_set, [], [], timeout)[0]:
        data = os.read(pipe.fileno(), 1<<30)
        if data:
            print("got from %s: %r" % (pipename[pipe], data))
        else: # EOF
            pipe.close()
            read_set.remove(pipe)
print("exit code %s" % (p.wait(),))

# child exited, wait for grandchild to print
for pipe in read_set:
    print("read the rest of %s: %r" % (pipename[pipe], pipe.read()))
    pipe.close()

where cmd:

import sys
from textwrap import dedent

cmd = [sys.executable, '-u', '-c', dedent("""
    # inception style
    import os
    import sys
    from subprocess import Popen
    from textwrap import dedent

    Popen([sys.executable, '-u', '-c', dedent('''
        import os
        import sys
        import time

        time.sleep(60)
        print("grandchild %d done" % os.getpid())
        sys.stderr.write("grandchild stderr")
        sys.exit(20)
    ''')]) # stdout/stderr are not redirected

    print('child %d done' % os.getpid())
    sys.stderr.write('child stderr')
    sys.exit(19)
""")]

score 0 · Answer 2 · answered Mar 04 '14 at 13:12

I've tried a simple example. And all I've got so far is capturing of the std.err I've created two files. The contents of the first is (example1.py):

import time
import sys

for i in xrange(1000):
    print >>sys.stderr, "hello", i
    time.sleep(1)

The contents of the second one(example2.py):

import subprocess

cmd = "python example1.py"

print subprocess.check_output(cmd, shell=True)

And that I've just called the example2.py script:

python example2.py And I've realtime output ( I assume it is true:) ):

hello 0
hello 1
hello 2
hello 3
hello 4
hello 5
hello 6

Still, I have no idea, how to deal with standard output, but if I manage it, I will post the answer here

How can I call a command using Python, capturing stderr and stdout, without waiting for stderr/stdout to be closed?

2 Answers2