20

I'm running a script that executes a number of executables by using 

subprocess.call(cmdArgs,stdout=outf, stderr=errf)

when outf/errf is either None or a file descriptor (different files for stdout/stderr).

Is there any way I can execute each exe so that the stdout and stderr will be written to the files and terminal together?

kiri
  • 2,522
  • 4
  • 26
  • 44
user515766
  • 349
  • 2
  • 5
  • 7

2 Answers2

28

The call() function is just Popen(*args, **kwargs).wait(). You could call Popen directly and use stdout=PIPE argument to read from p.stdout:

#!/usr/bin/env python
import sys
from subprocess import Popen, PIPE
from threading import Thread


def tee(infile, *files):
    """Print `infile` to `files` in a separate thread."""

    def fanout(infile, *files):
        with infile:
            for line in iter(infile.readline, b""):
                for f in files:
                    f.write(line)

    t = Thread(target=fanout, args=(infile,) + files)
    t.daemon = True
    t.start()
    return t


def teed_call(cmd_args, **kwargs):
    stdout, stderr = [kwargs.pop(s, None) for s in ["stdout", "stderr"]]
    p = Popen(
        cmd_args,
        stdout=PIPE if stdout is not None else None,
        stderr=PIPE if stderr is not None else None,
        **kwargs
    )
    threads = []
    if stdout is not None:
        threads.append(
            tee(p.stdout, stdout, getattr(sys.stdout, "buffer", sys.stdout))
        )
    if stderr is not None:
        threads.append(
            tee(p.stderr, stderr, getattr(sys.stderr, "buffer", sys.stderr))
        )
    for t in threads:
        t.join()  # wait for IO completion
    return p.wait()


outf, errf = open("out.txt", "wb"), open("err.txt", "wb")
assert not teed_call(["cat", __file__], stdout=None, stderr=errf)
assert not teed_call(["echo", "abc"], stdout=outf, stderr=errf, bufsize=0)
assert teed_call(["gcc", "a b"], close_fds=True, stdout=outf, stderr=errf)
jfs
  • 399,953
  • 195
  • 994
  • 1,670
  • thanks, what would you do if instead of subprocess.Call I'de like to run multiple execs using subprocess.Popen (and not Call), where each exec writes to a different file and to the terminal – user515766 Feb 14 '11 at 09:02
  • 1
    @user515766: the solution is the same: set `stdout`, `stderr` to `PIPE` and call `tee()` when you'd like to write to more than one place. – jfs Feb 14 '11 at 15:49
  • thanks for the quick response, but it doesn't work. the external process only sees OS-level file handles (the number you get from the fileno() method on your file objects). see http://bytes.com/topic/python/answers/541085-extend-file-type – user515766 Feb 13 '11 at 14:00
  • 1
    somebody has deleted comments that demonstrate that the first comment ("doesn't work") is wrong. It confuses`subprocess.call` and the function `call` (different) that is called `teed_call` now to avoid the ambiguity. – jfs Oct 24 '18 at 17:35
  • @Peilonrayz: I've made the code in the answer to be Python 2/3 compatible. (it was pure Python 2 solution in 2011) – jfs Dec 15 '19 at 11:24
  • I had to add this to get the example to work: `if isinstance(f, io.TextIOBase): f.write(str(line)) else: f.write(line)` – channon Nov 03 '20 at 17:58
  • @channon: `str` is likely a wrong thing to do. Notice, the code in the answer open files in *binary* mode. If you want support text files, specify an appropriate for your case character encoding such as utf-8, cp437, etc (or pass `text=True` if you want the `subprocess` module to chose it for you)$ Also, `sys.stdout.buffer` instead of `sys.stdout` should be used in binary mode on Python 3 (perhaps, this part causes the error) – jfs Nov 03 '20 at 18:22
  • Yes you are correct, the better way is to use the`sys.stdout.buffer`. Now the test cases provided pass as expected. – channon Nov 03 '20 at 18:42
0

You could use something like this: https://github.com/waszil/subpiper

In your callbacks you can do whatever you like, log, write to file, print, etc. It also supports non-blocking mode.

from subpiper import subpiper

def my_stdout_callback(line: str):
    print(f'STDOUT: {line}')

def my_stderr_callback(line: str):
    print(f'STDERR: {line}')

my_additional_path_list = [r'c:\important_location']

retcode = subpiper(cmd='echo magic',
                   stdout_callback=my_stdout_callback,
                   stderr_callback=my_stderr_callback,
                   add_path_list=my_additional_path_list)
waszil
  • 390
  • 2
  • 5
  • 15
  • I tried this with cmd='python3 run2.py' but get an error. Am I missing something how to send run2.py as a parameter: FileNotFoundError: [Errno 2] No such file or directory: 'python3 run2.py': 'python3 run2.py' – Rimfire Aug 12 '19 at 10:56