Python redirect subprocess to file and to console

Question

Using Python, I want to create a subprocess and have its output redirected to both a file AND the console.

I found this post explaining how to print to both console and file, but solutions does not work when creating a subprocess:

sys.stdout = Logger()
print( "Hello") # printed to console and file
res = subprocess.call(cmd) # command output is printed to console only

Same behaviour here:

with Tee('outfile.log', 'w'):
    print( "Hello" )
    res = subprocess.call(cmd)

How can I redirect subprocess output both to console (for user) and to file (for me to check it from my code).

Note: Im' on Windows, so using system's tee is not appropriate.

@hlt: Tried that, but does not work, it reports errors about Logger not being of the proper type.... — jpo38, Jul 23 '15 at 09:23

falsetru · Answer 1 · 2015-07-24T00:55:00.597

4

Connect cmd's standard output to tee's stdin:

# cmd = ['/bin/echo', 'blah blah']
tee = subprocess.Popen(['/usr/bin/tee', 'outfile.log'], stdin=subprocess.PIPE)
subprocess.call(cmd, stdout=tee.stdin)
tee.stdin.close()

UPDATE For systems that does not have tee, read the output of the command and write it into stdout, and the log file:

proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                        universal_newlines=True)

with open('outfile.log', 'w') as f:
    while True:
        data = proc.stdout.read(1024)
        if not data:
            break
        sys.stdout.write(data)
        f.write(data)

edited Jul 24 '15 at 00:55

answered Jul 23 '15 at 09:23

falsetru

357,413
63
732
636

I'm on Windows. Need a more generic solution. Sorry, I'll update OP. – jpo38 Jul 23 '15 at 09:24
@jpo38, I updated the answer accordingly. – falsetru Jul 23 '15 at 09:29
Tried it, got an error `sys.stdout.write(data)` `TypeError: must be str, not bytes`. Tried `str(data)` but then `\r\n` are being displayed and output comes on a single line.... – jpo38 Jul 23 '15 at 12:30
Moreover, it's hard to decide what read parameter should be, 1024 may be way to much for processes writing few data. The `while True` loop may change the timing in which process output comes to the console... – jpo38 Jul 23 '15 at 12:32
Instead of using str(data), use `data.decode("utf-8")` or `data.decode("ascii")` – muddyfish Jul 23 '15 at 12:49
Works....but only for `stdout`, how to handle `stderr` to have ALL outputs!? – jpo38 Jul 23 '15 at 12:59
As shown in the code, by specifying `stderr=subprocess.STDOUT`, you will get standard error output with standard output. – falsetru Jul 23 '15 at 14:55
plus one for the 2nd code example. You should close `proc.stdout` in the 1st one otherwise `cmd` might hang if `tee` dies. Or use `tee = Popen(['tee', 'logfile'], stdin=PIPE); call(cmd, stdout=tee.stdin)` See [How do I use subprocess.Popen to connect multiple processes by pipes?](http://stackoverflow.com/a/9164238/4279) – jfs Jul 23 '15 at 23:19
@J.F.Sebastian, Thank you for the comment. Closing `proc.stdout` seems inappropriate; closing before `call` will cause IOError. closing afterward- no benefit. I updated the answer to use your second suggestion. – falsetru Jul 24 '15 at 00:50
@falsetru: 1. obviously, you would use `Popen()` instead of `call()`, to close the pipe (`p.stdout`) in time. 2. You could call `tee.communicate()` after the `call()` – jfs Jul 24 '15 at 00:55
@eryksun, Thank you for the comment, I updated the answer accordingly. I didn't know about `universal_newlines`. – falsetru Jul 24 '15 at 00:55

Python redirect subprocess to file and to console

1 Answers1