Filtering log levels

Question

I am trying to separate the log levels into separate files (one for each level). At the moment I have defined a file for each level but with my current configuration the upper levels are propagated to the lower levels.

My log configuration is:

version: 1

formatters:
  standard:
    format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
  error:
    format: "%(levelname)s <PID %(process)d:%(processName)s> %(name)s.%(funcName)s(): %(message)s"

handlers:
  console:
    class: logging.StreamHandler
    formatter: standard
    level: DEBUG

  debug_file_handler:
    class: logging.handlers.RotatingFileHandler
    formatter: standard
    level: DEBUG
    filename: logs/debug.log
    encoding: utf8
    mode: "w"
    maxBytes: 10485760 # 10MB
    backupCount: 1

  info_file_handler:
    class: logging.handlers.RotatingFileHandler
    formatter: standard
    level: INFO
    filename: logs/info.log
    encoding: utf8
    mode: "w"
    maxBytes: 10485760 # 10MB
    backupCount: 1

  warning_file_handler:
    class: logging.handlers.RotatingFileHandler
    formatter: standard
    level: WARNING
    filename: logs/warning.log
    encoding: utf8
    mode: "w"
    maxBytes: 10485760 # 10MB
    backupCount: 1

  error_file_handler:
    class: logging.handlers.RotatingFileHandler
    formatter: error
    level: ERROR
    filename: logs/error.log
    encoding: utf8
    mode: "w"
    maxBytes: 10485760 # 10MB
    backupCount: 1

  critical_file_handler:
    class: logging.handlers.RotatingFileHandler
    formatter: error
    level: CRITICAL
    filename: logs/critical.log
    encoding: utf8
    mode: "w"
    maxBytes: 10485760 # 10MB
    backupCount: 1

loggers:
  development:
    handlers: [ console, debug_file_handler ]
    propagate: false
  production:
    handlers: [ info_file_handler, warning_file_handler, error_file_handler, critical_file_handler ]
    propagate: false
root:
  handlers: [ debug_file_handler, info_file_handler, warning_file_handler, error_file_handler, critical_file_handler ]

And I load the configuration and set the logger like this:

with open(path_log_config_file, 'r') as config_file:
    config = yaml.safe_load(config_file.read())
    logging.config.dictConfig(config)

logger = logging.getLogger(LOGS_MODE)
logger.setLevel(LOGS_LEVEL)

Where LOGS_MODE and LOGS_LEVEL are defined in a configuration file in my project:

# Available loggers: development, production
LOGS_MODE = 'production'
# Available levels: CRITICAL = 50, ERROR = 40, WARNING = 30, INFO = 20, DEBUG = 10
LOGS_LEVEL = 20

And when I want to use the logger I do:

from src.logger import logger

I have found these answers where they mention to use filters: #1 #2 but both of them say to use different handlers and specify the level for each one but with this approach I'll have to import different loggers in some cases instead of only one. Is this the only way to achieve it?

Regards.

UPDATE 1:

As I am using a YAML file to load the logger configuration I found this answer #3:

So I have defined the filters in my file logger.py:

with open(path_log_config_file, 'rt') as config_file:
    config = yaml.safe_load(config_file.read())
    logging.config.dictConfig(config)
    

class InfoFilter(logging.Filter):
    def __init__(self):
        super().__init__()

    def filter(self, record):
        return record.levelno == logging.INFO


class WarningFilter(logging.Filter):
    def __init__(self):
        super().__init__()

    def filter(self, record):
        return record.levelno == logging.WARNING


class ErrorFilter(logging.Filter):
    def __init__(self):
        super().__init__()

    def filter(self, record):
        return record.levelno == logging.ERROR


class CriticalFilter(logging.Filter):
    def __init__(self):
        super().__init__()

    def filter(self, record):
        return record.levelno == logging.CRITICAL


logger = logging.getLogger(LOGS_MODE)
logger.setLevel(LOGS_LEVEL)

And in the YAML file:

filters:
  info_filter:
    (): src.logger.InfoFilter
  warning_filter:
    (): src.logger.WarningFilter
  error_filter:
    (): src.logger.ErrorFilter
  critical_filter:
    (): src.logger.CriticalFilter

handlers:
  console:
    class: logging.StreamHandler
    formatter: standard
    level: DEBUG

  debug_file_handler:
    class: logging.handlers.RotatingFileHandler
    formatter: standard
    level: DEBUG
    filename: logs/debug.log
    encoding: utf8
    mode: "w"
    maxBytes: 10485760 # 10MB
    backupCount: 1

  info_file_handler:
    class: logging.handlers.RotatingFileHandler
    formatter: standard
    level: INFO
    filename: logs/info.log
    encoding: utf8
    mode: "w"
    maxBytes: 10485760 # 10MB
    backupCount: 1
    filters: [ info_filter ]

  warning_file_handler:
    class: logging.handlers.RotatingFileHandler
    formatter: standard
    level: WARNING
    filename: logs/warning.log
    encoding: utf8
    mode: "w"
    maxBytes: 10485760 # 10MB
    backupCount: 1
    filters: [ warning_filter ]

  error_file_handler:
    class: logging.handlers.RotatingFileHandler
    formatter: error
    level: ERROR
    filename: logs/error.log
    encoding: utf8
    mode: "w"
    maxBytes: 10485760 # 10MB
    backupCount: 1
    filters: [ error_filter ]

  critical_file_handler:
    class: logging.handlers.RotatingFileHandler
    formatter: error
    level: CRITICAL
    filename: logs/critical.log
    encoding: utf8
    mode: "w"
    maxBytes: 10485760 # 10MB
    backupCount: 1
    filters: [ critical_filter ]

My problem now is in the filter section. I don't know how to specify the name of each class. In the response #3 he uses __main__. because he is running the script directly, not as a module and doesn't says how to do it if you use a module.

Reading the User-defined objects doc reference I've tried to use ext:// as it's said in the Access to external objects section but I get the same error as when trying to specify the hierarchy with src.logger.InfoFilter.

    logging.config.dictConfig(config)
  File "/usr/lib/python3.8/logging/config.py", line 808, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/lib/python3.8/logging/config.py", line 553, in configure
    raise ValueError('Unable to configure '
ValueError: Unable to configure filter 'info_filter'
python-BaseException

My project tree is (only the important part is shown):

.
├── resources
│   ├── log.yaml
│   └── properties.py
├── src
│   ├── main.py
│   └── logger.py
└── ...

Answer 1

I submit another answer because your question changed considerably with your update 1.

Notes on replication :

• I recreated your arborescence
• my PYTHONPATH only pointed to the root (parent of src/ and ressources/)
• I ran the script from the root (current directory)
• I created a logs/ directory at the top-level (otherwise I got ValueError: Unable to configure handler 'critical_file_handler': [Errno 2] No such file or directory: 'C:\\PycharmProjects\\so69336121\\logs\\critical.log')

The problem you encountered was caused by cyclical imports. When the logger module was imported, it started by loading and YAML file, which asked to instantiate some src.logger.*Filter objects, which it could not found because the file was not yet fully initialized. I recommend putting effectful code into functions that can be called by the main function at startup.

Here is what I have :

# file: src/logger.py

import logging.config

import yaml  # by `pip install pyyaml`


path_log_config_file = "ressources/log.yml"
LOGS_LEVEL = logging.ERROR
LOGS_MODE = "production"


def setup_logging():
    with open(path_log_config_file, 'rt') as config_file:
        config = yaml.safe_load(config_file.read())
        logging.config.dictConfig(config)

# ... the rest of the file you provided

# file: src/main.py

from src.logger import setup_logging, logger


setup_logging()

logger.debug("DEBUG")
logger.info("INFO")
logger.warning("WARNING")
logger.error("ERROR")
logger.critical("CRITICAL")

Then I got an error :

ValueError: dictionary doesn't specify a version

solved by adding this line tot he top of the YAML file :

version: 1

cf documentation

Then I got this error :

ValueError: Unable to configure handler 'console': Unable to set formatter 'standard': 'standard'

Because your formatters were not defined (you probably mis-copy-pasted). Here you go, add that to your YAML file :

formatters:
  standard:
    format: '%(asctime)s [%(levelname)s] %(name)s: %(message)s'
  error:
    format: 'ERROR %(asctime)s [%(levelname)s] %(name)s: %(message)s'

It ran with no error, but nothing was written to the logs. A quick debugger breakpoint showed that the *Filter.filter methods were never called. I examined the logger object and indeed it had no handler attached. It can be added to the YAML too :

loggers:
  production:
    handlers: [ debug_file_handler, info_file_handler, warning_file_handler, error_file_handler, critical_file_handler ]
    propagate: False

And now it works.

Answer 2

I think you misunderstood.

both of them say to use different handlers and specify the level for each one

Correct.

but with this approach I'll have to import different loggers in some cases instead of only one

No, you can add as many handlers as you want to one logger. That's why the method is called Logger.addHandler and that each logger object has a list of handlers (its .handlers member).

You only need to have one logger setup with your 5 handlers.