Logging every exception within custom Python package

Question

I'm working on implementing logging into a custom Python package. Throughout the package, exceptions can be raised if inputs are unacceptable, or if tests fail, etc. I'd like to log any exceptions that are raised within the package using the built-in logging module.

So far, I've seen a few solutions. The simplest is wrapping any code that could raise an Exception with a try-catch block:

import logging

try:
    raise Exception
except Exception as e:
    logging.error(e)
    raise

This would be acceptable if there were only a few places where assert is used or exceptions can be otherwise be thrown within the package. At the scale of the package, wrapping every line that could raise an Exception is way too burdensome.

I've also seen solutions involving overriding sys.excepthook with a custom function (see here), which seems to work decently. However, setting sys.excepthook will cause any Exception raised by an end-user's code to also log all errors, which is not ideal.

# Some end-user's program
import my_custom_package
raise Exception  # This gets logged!

Ideally, I want a solution that logs every exception that arises strictly within the package, without modifying lines that would raise Exceptions, and without changing global behavior that affects end-users.

IMPLEMENTED SOLUTION:

# __init__.py

# Set up logging
logger = logging.getLogger(__name__)

# Override exception handler
def _exception_handler(exc_type, exc_value, exc_traceback):
    # Handle exception normally if it's a KeyboardInterrupt
    if issubclass(exc_type, KeyboardInterrupt):
        sys.__excepthook__(exc_type, exc_value, exc_traceback)
        return
    # Handle exception normally if error originates outside of library
    exc_filepath = Path(traceback.extract_tb(exc_traceback)[-1].filename)
    if not Path(__file__).parent in exc_filepath.parents:
        sys.__excepthook__(exc_type, exc_value, exc_traceback)
        return
    # Log error if it comes from library
    logger.error("uncaught exception", exc_info=(
        exc_type, exc_value, exc_traceback))
sys.excepthook = _exception_handler

score 1 · Accepted Answer · answered Jun 30 '22 at 00:27

Your exception hook can test to see if the exception is being raised in your package by seeing if it's one of your exceptions (use issubclass() to see whether the passed-in exception type is derived from your package's exception base class) or comes from one of your modules (check the passed-in traceback object; each stack frame's code object contains the filename in which the function was defined).

However, this only logs unhandled exceptions. If you want to log all exceptions, even if they are handled, the best place to do that is in the __init__ method of your exception base class. You'll need to use the inspect module to identify who is raising the exception since it hasn't actually been raised at that point, but it's doable.

Logging every exception within custom Python package

1 Answers1