Capturing subprocess in console and python

Question

I'm trying to execute some bash command using python. I want to display command's live output to user as well as capture it.
A sample example is like

import subporcess

# This will store output in result but print nothing to terminal
result = subprocess.run(['ls', '-lh'], check=True, universal_newlines=True, stdout=subprocess.PIPE)
print(result.stdout) # STD OUTPUT
# This will print everything to terminal result will be empty
result = subprocess.run(['ls', '-lh'], check=True, universal_newlines=True)
print(result.stdout) # OUTPUT = None

instead of `.run` you can use `.check_output`. Have a look at the [documentation](https://docs.python.org/3/library/subprocess.html#subprocess.check_output) — Ma0, Jun 21 '21 at 13:26
If you will go through the function arguments you'll see that there is not stdout option. i.e checkout will only return output to python but print nothing on the console. — Pranjal Doshi, Jun 21 '21 at 13:39
Yes.. But The process I am trying to run takes around `30 min` till then the user will think that the program is dead. — Pranjal Doshi, Jun 21 '21 at 13:45
in that case you can subclass `subprocess.PIPE` and make it write the output in two places at the same time. — Ma0, Jun 21 '21 at 13:49

score 2 · Answer 1 · answered Jun 21 '21 at 13:55

Here is one possibility which will gather output lines from a long-running process, write them to terminal as it goes, and return them all when the process exits.

It returns a list of output lines, rather than a full block of text that check_output or run would return, but that's easily changed. Perhaps an IO buffer might be more efficient depending on how much output you're expecting.

import subprocess
import sys

def capture_and_echo_stdout(cmd):
    """ Start a subprocess and write its stdout to stdout of this process.     
    Capture and return stdout lines as a list. """
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
    stdout_lines = []
    for line in proc.stdout:
        sys.stdout.write(line.decode())
        stdout_lines.append(line)
    proc.communicate()
    # Roughly equivalent to check=True
    if proc.returncode:
        raise subprocess.CalledProcessError(proc.returncode, cmd)
    return stdout_lines

There are a few similar options in this answer (although the focus there is more on writing to multiple files like unix tee): How to replicate tee behavior in Python when using subprocess?

Thanks.. it is one of the option that I considered. But `run` is pretty stable and covers some corner cases which I won't be able to handle while using Popen. But thanks for great answer. — Pranjal Doshi, Jun 21 '21 at 14:04

Capturing subprocess in console and python

1 Answers1