How to use python logging in multiple modules

Question

I was wondering what the standard set up is for performing logging from within a Python app.

I am using the Logging class, and I've written my own logger class that instantiates the Logging class. My main then instantiates my logger wrapper class. However, my main instantiates other classes and I want those other classes to also be able to write to he log file via the logger object in the main.

How do I make that logger object such that it can be called by other classes? It's almost like we need some sort of static logger object to get this to work.

I guess the long and short of the question is: how do you implement logging within your code structure such that all classes instantiated from within main can write to the same log file? Do I just have to create a new logging object in each of the classes that points to the same file?

Answer 1

I don't know what you mean by the Logging class - there's no such class in Python's built-in logging. You don't really need wrappers: here's an example of how to do logging from arbitrary classes that you write:

import logging

# This class could be imported from a utility module
class LogMixin(object):
    @property
    def logger(self):
        name = '.'.join([__name__, self.__class__.__name__])
        return logging.getLogger(name)


# This class is just there to show that you can use a mixin like LogMixin
class Base(object):
    pass

# This could be in a module separate from B
class A(Base, LogMixin):
    def __init__(self):
        # Example of logging from a method in one of your classes
        self.logger.debug('Hello from A')

# This could be in a module separate from A
class B(Base, LogMixin):
    def __init__(self):
        # Another example of logging from a method in one of your classes
        self.logger.debug('Hello from B')

def main():
    # Do some work to exercise logging
    a = A()
    b = B()
    with open('myapp.log') as f:
        print('Log file contents:')
        print(f.read())

if __name__ == '__main__':
    # Configure only in your main program clause
    logging.basicConfig(level=logging.DEBUG,
                        filename='myapp.log', filemode='w',
                        format='%(name)s %(levelname)s %(message)s')
    main()

Generally it's not necessary to have loggers at class level: in Python, unlike say Java, the unit of program (de)composition is the module. However, nothing stops you from doing it, as I've shown above. The script, when run, displays:

Log file contents:
__main__.A DEBUG Hello from A
__main__.B DEBUG Hello from B

Note that code from both classes logged to the same file, myapp.log. This would have worked even with A and B in different modules.

Answer 2

Try using logging.getLogger() to get your logging object instance:

http://docs.python.org/3/library/logging.html#logging.getLogger

All calls to this function with a given name return the same logger instance. This means that logger instances never need to be passed between different parts of an application.

UPDATE:

The recommended way to do this is to use the getLogger() function and configure it (setting a handler, formatter, etc...):

# main.py
import logging
import lib


def main():
    logger = logging.getLogger('custom_logger')
    logger.setLevel(logging.INFO)
    logger.addHandler(logging.FileHandler('test.log'))
    logger.info('logged from main module')
    lib.log()

if __name__ == '__main__':
    main()

# lib.py
import logging


def log():
    logger = logging.getLogger('custom_logger')
    logger.info('logged from lib module')

If you really need to extend the logger class take a look at logging.setLoggerClass(klass)

UPDATE 2:

Example on how to add a custom logging level without changing the Logging class:

# main.py
import logging
import lib


# Extend Logger class
CUSTOM_LEVEL_NUM = 9
logging.addLevelName(CUSTOM_LEVEL_NUM, 'CUSTOM')
def custom(self, msg, *args, **kwargs):
    self._log(CUSTOM_LEVEL_NUM, msg, args, **kwargs)
logging.Logger.custom = custom

# Do global logger instance setup
logger = logging.getLogger('custom_logger')
logger.setLevel(logging.INFO)
logger.addHandler(logging.FileHandler('test.log'))


def main():
    logger = logging.getLogger('custom_logger')
    logger.custom('logged from main module')
    lib.log()

if __name__ == '__main__':
    main()

Note that adding custom level is not recommended: http://docs.python.org/2/howto/logging.html#custom-levels

Defining a custom handler and maybe using more than one logger may do the trick for your other requirement: optional output to stderr.