Capture output from non-python third-party library with python logger

Question

I have written a Python package that uses the logging module, and makes extensive use of a third-party C++ library with a Python wrapper. I have been able to print messages within my own package to the console, as well as write them to a file (and with different logging levels for each handler). However, I would like to include messages printed by the third-party library in my log file so that I see the order in which they occur. Here is my MWE:

import logging

# Assume no direct access to this function. (e.g. c++ library)
def third_party_function():
    print("Inside of 'third_party_function'.")

def my_helper():
    logger.debug("Inside of 'my_helper', before third party call.")
    third_party_function()
    logger.warning("Finished with third party call.")

root_logger = logging.getLogger()
root_logger.setLevel(logging.NOTSET)

logger = logging.getLogger("mylogger")

stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.WARNING)

file_handler = logging.FileHandler(filename="progress.out")
file_handler.setLevel(logging.NOTSET)

logger.addHandler(stream_handler)
logger.addHandler(file_handler)

my_helper()

As it stands, the output to the screen is:

Inside of 'third_party_function'.
Finished with third party call.

and the file progress.out contains

Inside of 'my_helper', before third party call.
Finished with third party call.

However, the desired progress.out file is

Inside of 'my_helper', before third party call.
Inside of 'third_party_function'.
Finished with third party call.

There doesn't seem to be a logger that belongs to this third-party library since it is not written in Python.

I am hoping to avoid setting sys.stdout to a file (as found here) since I would like to be consistent and use the logging module throughout. Another answer for that same question defines a custom class, but this still doesn't seem to catch the third-party message.

kielni · Accepted Answer · 2020-09-19T20:32:48.497

You can temporarily redirect stdout from your 3rd party function go to a logger.

Using the StreamToLogger class from the answer you linked: Redirect Python 'print' output to Logger

def wrap_third_party_function():
    stdout = sys.stdout # save for restoring afterwards
    # redirect stdout to use a logger, with INFO level
    sys.stdout = StreamToLogger(logging.getLogger("mylogger.thirdparty"), logging.INFO)
    print("start third_party_function; stdout to log")
    third_party_function()
    sys.stdout = stdout  # restore stdout
    print("done third_party_function; stdout restored")

def my_helper():
    logger.debug("Inside of 'my_helper', before third party call.")
    wrap_third_party_function()
    logger.warning("Finished with third party call.")

On the console, you'll get this:

done third_party_function; stdout restored
Finished with third party call.

And progress.out will have:

Inside of 'my_helper', before third party call.
start third_party_function; stdout to log
Inside of 'third_party_function'.
Finished with third party call.

Adding a formatter makes it easier to see:

formatter = logging.Formatter("%(name)s %(levelname)-8s %(message)s")
stream_handler.setFormatter(formatter)
file_handler.setFormatter(formatter)

With this configuration, progress.out shows:

mylogger DEBUG    Inside of 'my_helper', before third party call.
mylogger.thirdparty INFO     start third_party_function; stdout to log
mylogger.thirdparty INFO     Inside of 'third_party_function'.
mylogger WARNING  Finished with third party call.

Capture output from non-python third-party library with python logger

1 Answers1