Python logger in the hierarchy under root and over my project loggers?

Question

First of all, I have read real python article on the subject.

Having learnt that loggers have a hierarchy, I want to create this new one called MyProjectLogger in such a hierarchy:

Root logger
    · MyProjectLogger
         · File logger 1
         · File logger 2
         · ... and so on with all the loggers in my project...

so that MyProjectLogger is used for all the descendant loggers, because right now I'm using all the same handlers with same configuration in all the loggers in my project (quite a lot). Although doing it through an only method, it doesn't feel right. In this way I would add the handlers only once to MyProjectLogger and all descendant loggers would just go up in the hierarchy using MyProjectLogger's handlers.

I don't want to use the default root logger for this because I have some third party libraries which are logging on it and right now I want the loggers in my project to log separately from the loggers in the libraries.

So, in summary:

I want to define a logger MyProjectLogger in the hierarchy
I want it to be direct descendant of root logger
I want it to be the parent of all the loggers in my source code
I believe I should use propagate=False so I can add the handlers to MyProjectLogger and getting it to handle of descendant loggers

My only doubt is: how do I give it such a name so that is under root and over the rest?

I know that:

logging.getLogger()  # Gets the root logger
logging.getLogger(__name__)  # Gets a logger for the present file
logging.getLogger(__package__)  # Gets a logger for the present module

so let's say, if my project has this folder layout:

aaaBot/
   main.py  # Only file visible in this example.
            # Please assume the rest of folders have files
   common/
      utils/ 
   config/
   database/
   exceptions/
   model/
   wizards/

In every file for each folder I use logging.getLogger(__name__). __package__ in the root returns None and in the main executable main.py __name__ is '__main__'.

Should I add a prefix + '.' for all the loggers in my project and create MyProjectLogger with that prefix (like getLogger(prefix+'.'))?

If not, what should I do?

Should `MyProjectLogger` be the desired name? Because `logging.getLogger('aaaBot')` is already the root logger in your project's hierarchy. — hoefling, Jul 14 '19 at 11:29
So, if I do logging.getLogger('aaaBot'), then every one created with logging.getLogger(____name____) will be under aaaBot logger? — madtyn, Jul 14 '19 at 11:31
Exactly. If you e.g. configure a logger in `aaaBot/__init__.py` with `root = logging.getLogger(__name__)`, then the parent of `logging.getLogger(__name__)` in e.g. `aaaBot/common/utils.py` will be `root`. The logger name hierarchy is derived from package/module name hierarchy for that reason, so you don't have to declare parent-child relationship explicitly. — hoefling, Jul 14 '19 at 11:46
It's also mentioned in the docs, check out the section under [Logger Objects](https://docs.python.org/3/library/logging.html#logger-objects). — hoefling, Jul 14 '19 at 11:50
I'm outside, but I'm testing this at night here (in about 10h). You're welcome to post this answer and as soon I check it, I'm accepting it. — madtyn, Jul 14 '19 at 12:58
I'm afraid that ___init___.py is not being called and I don't know how it should be called. Maybe my knowledge about __init__.py here is not enough. I'm reading the official python docs on the subject right now. — madtyn, Jul 15 '19 at 00:22
I was reading on the subject. It seems like that thing would be ok for developing a library, but my project is an application. I'm executing main.py and I don't know how an ___init___.py in the root folder would be called without importing it. — madtyn, Jul 15 '19 at 00:58
`__init__.py` will be called automatically when you import `aaaBot`. _It seems like that thing would be ok for developing a library_ - the hierarchy works for libraries and apps, the only difference is that you shouldn't do any logging configuration for libraries. I will write an answer with a working example to clarify. — hoefling, Jul 15 '19 at 10:46

hoefling · Answer 1 · 2019-07-15T11:38:12.760

Here's a working example illustrating the logger hierarchy mimicking the module structure:

so-57021706
└── aaaBot
    ├── __init__.py
    ├── common
    │   ├── __init__.py  # empty
    │   └── utils.py
    └── main.py

Source

aaaBot/__init__.py:

import logging
import sys


PKG_LOGGER = logging.getLogger(__name__)


def setup_logging():
    msg_format = '%(asctime)s [%(levelname)8s] %(message)s (%(name)s - %(filename)s:%(lineno)s)'
    date_format = '%Y-%m-%d %H:%M:%S'
    formatter = logging.Formatter(fmt=msg_format, datefmt=date_format)
    console_handler = logging.StreamHandler(stream=sys.stdout)
    console_handler.setLevel(logging.DEBUG)
    console_handler.setFormatter(formatter)
    PKG_LOGGER.addHandler(console_handler)
    PKG_LOGGER.setLevel(logging.DEBUG)
    PKG_LOGGER.propagate = False
    PKG_LOGGER.info('finished logging setup!')

aaaBot/common/utils.py:

import logging


UTILS_LOGGER = logging.getLogger(__name__)


def spam():
    UTILS_LOGGER.debug('entered spam() function')
    output = 'eggs'
    UTILS_LOGGER.debug('leaving spam() function')
    return output

aaaBot/main.py:

import sys
from aaaBot import setup_logging
from aaaBot.common.utils import spam


if __name__ == '__main__':
    if sys.argv[-1] == '--with-logging':
        setup_logging()
    print(spam())

Execution

Normal run:

$ python -m aaaBot.main
eggs

Debug run (turns logging on):

$ python -m aaaBot.main --with-logging
2019-07-15 13:16:04 [    INFO] finished logging setup! (aaaBot - __init__.py:18)
2019-07-15 13:16:04 [   DEBUG] entered spam() function (aaaBot.common.utils - utils.py:8)
2019-07-15 13:16:04 [   DEBUG] leaving spam() function (aaaBot.common.utils - utils.py:10)
eggs

Explanation

In this example project, the PKG_LOGGER under aaaBot/__init__.py is the "project" logger, having the name aaaBot. It's also the one and only logger that is configured; all the child loggers do nothing else than propagate the records up to PKG_LOGGER. The example of a child logger is UTILS_LOOGER from aaaBot/common/utils.py - not configured and having the name aaaBot.common.utils. The hierarchy in this case is:

root logger           code: logging.getLogger()
                      name: "root"
                      configured: no
                      propagates to parent: no (already root)
└── PKG_LOGGER        code: logging.getLogger('aaaBot')
                      name: "aaaBot"
                      configured: yes
                      propagates to parent: no (because of propagate = False)
    └── UTILS_LOGGER  code: logging.getLogger('aaaBot.common.utils')
                      name: "aaaBot.common.utils"
                      configured: no
                      propagates to parent: yes (default behaviour)

The configuration possibilities are endless and depend on your particular use case. For example, you can configure the root logger only and make sure all loggers propagate (they do that by default anyway). This will also print all messages from third-party libraries you are using, should they want to log anything. Or you can introduce an additional logger aaaBot.common that will write records from child loggers to file, in addition to propagating and emitting them to console etc.

Regarding your comment:

It seems like that thing would be ok for developing a library, but my project is an application.

It doesn't matter; it only depends on the value of the __name__ variable (importing a module versus executing it). For example, using logging.getLogger(__name__) in aaaBot/main.py will not create a logger with the name aaaBot.main with aaaBot as parent. Instead, a logger named __main__ will be created.

The difference in logging setup between a library and an application is only the logger configuration. An application always configures logging explicitly if in need of logging. A library doesn't configure any logging at all, except a NullHandler to the library's root logger. For example, if aaaBot would be a library, the logging setup in aaaBot/__init__.py could look like this:

import logging
LIB_LOGGER = logging.getLogger('aaaBot')
if not LIB_LOGGER.handlers:
    LIB_LOGGER.addHandler(logging.NullHandler())

Is it relevant that so-57021706 folder? PyCharm is doing some werid things when autocompleting — madtyn, Jul 15 '19 at 14:19
No, not at all; it's a folder that is automatically created when I need to write some code for an answer. You can safely omit that. — hoefling, Jul 15 '19 at 14:21
However, you can think of `so-57021706` folder in the example as Pycharm's "source root" - the parent dir of the `aaaBot` package dir. — hoefling, Jul 15 '19 at 14:22
`ValueError: attempted relative import beyond top-level package` Seems like I need to nest everything one level, doesn't it? — madtyn, Jul 15 '19 at 14:32
It looks like you're using relative imports and the `sys.path` doesn't contain the needed directories. Check out [this answer](https://stackoverflow.com/a/47030746/2650249) and the ones linked in it. — hoefling, Jul 15 '19 at 14:38

Python logger in the hierarchy under root and over my project loggers?

1 Answers1

Source

Execution

Explanation