Python: Capture stdout from subprocess.call

Question

I'm trying to do two things when executing a shell cmd with Python:

Capture stdout and print it as it happens
Capture stdout as a whole and process it when the cmd is complete

I looked at subprocess.check_output, but it does not have an stdout param that would allow me to print the output as it happens.

So after reading this question, I realized I may need to try a different approach.

from subprocess import Popen, PIPE

process = Popen(task_cmd, stdout = PIPE)
stdout, stderr = process.communicate()

print(stdout, stderr)

The problem with this approach is that according to the docs, Popen.communicate():

Reads data from stdout and stderr, until end-of-file is reached. Wait for process to terminate

I still cannot seem to redirect output both to stdout AND to some sort of buffer that can be parsed when the command is complete.

Ideally, I'd like something like:

# captures the process output and dumps it to stdout in realtime
stdout_capture = Something(prints_to_stdout = True)
process = Popen(task_cmd, stdout = stdout_capture)

# prints the entire output of the executed process
print(stdout_capture.complete_capture)

Is there a recommended way to accomplish this?

Possible duplicate of [Retrieving the output of subprocess.call()](http://stackoverflow.com/questions/1996518/retrieving-the-output-of-subprocess-call) — Random Davis, Mar 07 '17 at 18:22
The other topics seem focused on simply capturing the output; whereas, I'm trying to both dump the output to stdout and capture it in its entirety for parsing on completion. I'm stuck at doing both simultaneously. — doremi, Mar 08 '17 at 17:24

score 2 · Answer 1 · answered Apr 26 '19 at 16:01

You were on the right track with using giving Popen stdout=PIPE, but you can't use .communicate() because it returns the values after execution. Instead, I suggest you read from .stdout.

The only guaranteed way to get the output the moment it's generated is to read from the pipe one character at a time. Here is my approach:

def passthrough_and_capture_output(args):
    import sys
    import subprocess

    process = subprocess.Popen(args, stdout=subprocess.PIPE, universal_newlines=True)
    # universal_newlines means that the output of the process will be interpreted as text
    capture = ""

    s = process.stdout.read(1)
    while len(s) > 0:
        sys.stdout.write(s)
        sys.stdout.flush()
        capture += s
        s = process.stdout.read(1)

    return capture

Note that reading one character at a time can incur significant overhead, so if you are alright with lagging behind a bit, I suggest that you replace the 1 in read(1) with a different number of characters to output in batches.

score -2 · Answer 2 · answered Jul 18 '18 at 17:57

from subprocess import check_output, CalledProcessError

def shell_command(args):
    try:
        res = check_output(args).decode()
    except CalledProcessError as e:
        res = e.output.decode()
    for r in ['\r', '\n\n']:
        res = res.replace(r, '')
    return res.strip()

Python: Capture stdout from subprocess.call

2 Answers2