Share Python logger across multiple files

Question

I have a python application with a file structure similar to the following:

/Project
    file1.py
    file2.py
    file3.py
    ...

The application is using Python 2.6 as it is running on a CentOS 6 environment.

I am interested in setting up a common logging mechanism where log statements from each of these files write to the same log file on disk. These python files are commonly executed from the command line are are somewhat independent of each other. One suggestion from the documentation (linked here) was the following which I have tested:

log = logging.getLogger(__name__)

What I found is that all log lines would be listed as coming from __main__, regardless of which file the log statement was from. In other words, I was not able to tell from which file a given log line came from.

I am hoping to get to a point were I can have a log file with contents similar to the following:

2017-01-17 10:48:47,446 - file1 - DEBUG - this is from file1
2017-01-17 10:48:47,447 - file2 - DEBUG - this is from file2
2017-01-17 10:48:47,447 - fiel3 - DEBUG - this is from file3

In the above, I would have executed:

log.debug("this is from file1")

in file1.py and a similar log line in the other files, replaces the text in the log statement as appropriate.

I am currently using the following to setup logging:

########################################
# Setup Logging
########################################
LOG_FILENAME = 'app.log'
# Set up a specific logger with our desired output level
log = logging.getLogger('Logger')
log.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# Add the log message handler to the logger
handler = logging.handlers.RotatingFileHandler(
              LOG_FILENAME, maxBytes=200000000, backupCount=5)
handler.setFormatter(formatter)
log.addHandler(handler)
########################################

This setup is all good for logging from one python source, but is less useful as I am now trying to log from multiple python sources

I have read Using Python logging in multiple modules; it has good answers, but due to the independent nature of the files in my project the use of __name__ was not working as described.

If you just want to use the filename rather than the module name, `logging.getLogger(__file__)` would do it. — jonrsharpe, Jan 23 '17 at 20:33
I will try this. Two follow up questions: 1) While the log statements from each python file be written to the same log file? Is there any changes needed to my posted logging config that would be required to do this? 2) Will I need to do anything extra to just get the just the name of the file and not the whole path of the file? Or will `__file__` be just the filename without path? I am a bit rusty on some of my python :( — Andrew, Jan 23 '17 at 20:40
Ok. It looks like my existing setup configuration with the use of `__file__` did the trick. I will need to work a bit on cutting of the path from the file name but otherwise, this is what I was looking for. — Andrew, Jan 23 '17 at 21:09

xjcl · Answer 1 · 2020-12-07T01:02:45.903

Best approach

Rather than trying to set logger name to the filename, just specify %(module)s!

formatter = logging.Formatter('%(asctime)s - %(module)s - %(levelname)s - %(message)s')
                                             ^^^^^^^^^^

This will avoid printing __main__ and nicely give you:

2020-11-06 23:19:03,827 - file2 - INFO - This is from file2
2020-11-06 23:19:03,827 - file3 - INFO - This is from file3
2020-11-06 23:19:03,827 - file1 - INFO - This is from file1

Enjoy!

PS: I am slightly nervous about you attaching a RotatingFileHandler of the same file to each logger, rather than to the root logger once. In general I wouldn't recommend copying logging-setup code to each source code file and logger, but configuring the root logger once in a separate file.

Share Python logger across multiple files

1 Answers1