Python: capturing both of sys.stdout and sys.stderr as a log file

Question

Roughly speaking, I want to port this to pure Python:

#!/bin/bash

{
    python test.py
} &> /tmp/test.log

This didn't work for some unknown reasons:

import os.path, sys
import tempfile

with open(os.path.join(tempfile.gettempdir(), "test.log"), "a") as fp:
    sys.stdout = sys.stderr = fp
    raise Exception("I'm dying")

The resulting test.log was empty (and I didn't see anything on my console,) when I tested it with Python 2.6.6, Python 2.7.8 and Python 3.4.2 on CentOS x86_64.

But Ideally I'd like a solution for Python 2.6.

(For now, it's tolerable to clutter the log file with intermixed output from stdout and stderr or multithreading, as long as any data won't simply disappear into a blackhole.)

Show me a concise and portable solution which is confirmed to work with an exception stack trace on sys.stderr. (Preferably something other than os.dup2)

Possible duplicate of [Redirect stdout to a file in Python?](http://stackoverflow.com/questions/4675728/redirect-stdout-to-a-file-in-python) — xrisk, Feb 04 '16 at 06:27
@Rishav That article surely gives me some general insights but I'm looking for a specific solution to my specific problem. — nodakai, Feb 04 '16 at 06:36
I works if instead of using `with` you just assign the file to `fp` — eli, Feb 04 '16 at 06:52
@eli You're right, so the stack trace is printed outside of `with`... — nodakai, Feb 04 '16 at 06:59

Uri Shalit · Answer 1 · 2016-02-04T07:07:12.647

2

You can use a method like this one:

import traceback
import sys
from contextlib import contextmanager


@contextmanager
def output_to_file(filepath, write_mode='w'):
    stdout_orig = None
    stderr_orig = None

    stdout_orig = sys.stdout
    stderr_orig = sys.stderr

    f = open(filepath, write_mode)

    sys.stdout = f
    sys.stderr = f

    try:
        yield
    except:
        info = sys.exc_info()
        f.write('\n'.join(traceback.format_exception(*info)))

    f.close()

    sys.stdout = stdout_orig
    sys.stderr = stderr_orig

And the the usage is:

with output_to_file('test.log'):
    print('hello')
    raise Exception('I am dying')

And the cat test.log produces:

hello
Traceback (most recent call last):

  File "<ipython-input-3-a3b702c7b741>", line 20, in outputi_to_file
    yield

  File "<ipython-input-4-f879d82580b2>", line 3, in <module>
    raise Exception('I am dying')

Exception: I am dying

edited Feb 04 '16 at 07:07

answered Feb 04 '16 at 06:49

Uri Shalit

2,198
2
19
29

In what sort of environment did you **confirm** `with output_to_file('test.log'): raise Exception("I'm dying")` worked? `test.log` was empy in my environment. – nodakai Feb 04 '16 at 06:54
You are right. didn't test for exceptions. Fixed it. See it now – Uri Shalit Feb 04 '16 at 07:07
Your approach is interesting but there's a problem with it; usually, when a thrown exception is not caught, the return code of the script is set to 1 rather than 0, indicating an abnormal completion of the script. But under your `try...catch...`, it won't happen. This can make a difference in shell scripting. – nodakai Feb 06 '16 at 20:19
So I got this variant, but it assumes an uncaught exception from the wrapped code block terminates the entire script. I think it kills half of the advantages of your approach. ```def output_to_file(filepath, write_mode='w'): stdout_orig = sys.stdout stderr_orig = sys.stderr with open(filepath, write_mode) as f: sys.stdout = f sys.stderr = f try: yield except: info = sys.exc_info() f.write('\n'.join(traceback.format_exception(*info))) sys.exit(1) finally: sys.stdout = stdout_orig sys.stderr = stderr_orig``` – nodakai Feb 06 '16 at 20:23
Another option is to reraise the exception (or use only finally depending on your need), and add a try except in your main function, doing one `sys.exit` for all your code. Keeping this function "clean" – Uri Shalit Feb 07 '16 at 01:45

score 2 · Answer 2 · answered Feb 04 '16 at 06:52

2

Remember that file objects are closed after with blocks :)

Use simply this:

sys.stdout = sys.stderr = open("test.log","w")
raise Exception("Dead")

Content of test.log after exit:

Traceback (most recent call last):
  File "test.py", line 5, in <module>
    raise Exception("Dead")
Exception: Dead

answered Feb 04 '16 at 06:52

xrisk

3,790
22
45

Hmmmm I confirmed there was data in `test.log` – nodakai Feb 04 '16 at 06:57
@nodakai what do you mean? – xrisk Feb 04 '16 at 07:02
So far it seems like a valid solution to my problem but I'd like to check if Python can ensure any data won't be lost. – nodakai Feb 04 '16 at 07:05

score 1 · Accepted Answer · answered Feb 04 '16 at 06:49

This works for me:

#!/usr/bin/env python
from __future__ import print_function
import os, os.path, sys, tempfile

old_out = os.dup(sys.stdout.fileno())
old_err = os.dup(sys.stderr.fileno())
with open(os.path.join(tempfile.gettempdir(), "test.log"), "a") as fp:
    fd = fp.fileno()
    os.dup2(fd, sys.stdout.fileno())
    os.dup2(fd, sys.stderr.fileno())
    print("Testing")
    print('testing errs', file=sys.stderr)
    raise Exception("I'm dying")

The future is just for cleaner handling of Python2 or Python3 with the same example. (I've also changed the raise statement to instantiate an exception, strings as exceptions have been deprecated for a long time and they're not properly supported under Python3).

The old_* values are just if we wanted to restore our original stdout and/or stderr after using our redirected file.

Thanks, `os.dup2` doesn't actually look like a bad solution – nodakai Feb 04 '16 at 07:01 — nodakai, Feb 04 '16 at 07:01

Python: capturing both of sys.stdout and sys.stderr as a log file

3 Answers3