How to asynchronously log stdout/stderr in Python?

Question

I have a function (which I cannot change) and it prints data to stdout/stderr. I need to log the output of this function. My first idea was to replace sys.stdout with a buffer in the form of StringIO() and then process its contents. That worked well, but the problem is, when the function fails, it prints the error message and exits the current process. In that case, contents of the buffer are lost, because my code after function call is never executed.

So my idea was to somehow watch the buffer asynchronously and process its contents immediatelly when there are data to be read. I tried a solution with asyncio and its add_reader method, but that seems to not support StringIO() and not even a regular file.

This is my first try to just asynchronously print the stdout:

import asyncio
import sys
from io import StringIO

async def f():
    print('Some output')

def logger(buffer):
    sys.__stdout__.write(buffer.read())

buffer = StringIO()
sys.stdout = buffer

loop = asyncio.get_event_loop()
loop.add_reader(buffer, logger, buffer)
loop.run_until_complete(f())

That fails with

ValueError: Invalid file object: <_io.StringIO object at 0x7f8a93e9aa68>

Is there any solution to this problem? At least I need to clarify if my approach makes sense.

UPDATE: I have discovered standard module atexit, which can call a function upon interpreter exit. This is another way to solve my problem.

Can you just stop the program from... eh... _stopping_ instead of monitoring the buffer? If there's an exception, you can just catch it and go on. — ForceBru, Mar 02 '18 at 14:55
@ForceBru The exception is processed inside the function, it will never return from the function call. — rubick, Mar 02 '18 at 15:05
Possible duplicate of [Making Python loggers output all messages to stdout in addition to log](https://stackoverflow.com/questions/14058453/making-python-loggers-output-all-messages-to-stdout-in-addition-to-log) — eyllanesc, Mar 02 '18 at 16:48

Sven Marnach · Accepted Answer · 2018-03-02T23:13:32.347

You can create a custom subclass of io.TextIOBase and replace sys.stdout with an instance of your custom class. The write() method of your class will be called whenever output is sent to sys.stdout. Optionally you can forward all output to the original stdout:

class MyStdOut(io.TextIOBase):
    def __init__(self, orig_stdout=None):
        self.orig_stdout = orig_stdout
    def write(self, s):
        # Process output in whatever way you like
        process(s)
        # Write output to original stream, if desired
        if self.orig_stdout:
            self.orig_stdout.write(s)

sys.stdout = MyStdOut(sys.stdout)

This aproach wil be completely synchronous – no threads or asynchronous I/O needed.

How to asynchronously log stdout/stderr in Python?

1 Answers1