How to use a callable as a filter in logging python

Question

I don't know how to use this new property of python3.2. There, instead of implementing logging.Filter class, one can use a callable.

I'm trying to use dictConfig for my logger (in python). In that, I want to add a filter such that it will pass if record's message contains certain phrase.
I know how to do that by implementing logging.Filter class.
But I don't know how to just use callable 'fancy' property of python 3.2 as stated here

Good code here

class ignore_progress(logging.Filter):
    def filter(self, record):

        return not ('Progress' in record.getMessage())
class log_progress(logging.Filter):
    def filter(self, record):
        return ('Progress' in record.getMessage())
def contain_progress(record):
    return not ('Progress' in record.message)
logging_dict = {
    "version": 1,
    "disable_existing_loggers": False,  
    "formatters": {
        "standard": {
            "format": "%(asctime)s [%(levelname)s] %(name)s: %(message)s",
        }
    },
    "filters": {
         "ignore_progress": {
            '()': ignore_progress,
        }
    },
    "handlers": {
        "default": {
            "class": "logging.StreamHandler",
            "level": "DEBUG",
            "formatter": "standard",
        },
        "file": {
            "class": "logging.FileHandler",
            "level": "DEBUG",
            "formatter": "standard",
            "filename": 'training_{}.log'.format(str(datetime.date.today())),
            "filters": ["ignore_progress"],
        },
    },
    "loggers": {
        "": {"handlers": ["default", "file"], "level": "DEBUG", "propagate": True, },
    },
}
# Configurate the logger
logging.config.dictConfig(logging_dict)
logger = logging.getLogger(__name__)

logger.info("Run training")
logger.info("Progress.test")

Bad code here

class ignore_progress(logging.Filter):
    def filter(self, record):

        return not ('Progress' in record.getMessage())
class log_progress(logging.Filter):
    def filter(self, record):
        return ('Progress' in record.getMessage())
def contain_progress(record):
    return not ('Progress' in record.message)
logging_dict = {
    "version": 1,
    "disable_existing_loggers": False,  
    "formatters": {
        "standard": {
            "format": "%(asctime)s [%(levelname)s] %(name)s: %(message)s",
        }
    },
    "filters": {
         "ignore_progress": {
            '()': contain_progress,
        }
    },
    "handlers": {
        "default": {
            "class": "logging.StreamHandler",
            "level": "DEBUG",
            "formatter": "standard",
        },
        "file": {
            "class": "logging.FileHandler",
            "level": "DEBUG",
            "formatter": "standard",
            "filename": 'training_{}.log'.format(str(datetime.date.today())),
            "filters": ["ignore_progress"],
        },
    },
    "loggers": {
        "": {"handlers": ["default", "file"], "level": "DEBUG", "propagate": True, },
    },
}
# Configurate the logger
logging.config.dictConfig(logging_dict)
logger = logging.getLogger(__name__)

logger.info("Run training")
logger.info("Progress.test")

The above bad code has a problem at this line in config.py

NoName · Answer 1 · 2021-02-23T17:54:50.717

When using Callable in dictConfig, the Callable you put into the value of dictConfig has to be a Callable which returns a Callable as discussed in the Python Bug Tracker:

https://bugs.python.org/issue41906

E.g.

def my_filter_wrapper():
    # the returned Callable has to accept a single argument (the LogRecord instance passed in this callable) with return value of 1 or 0
    return lambda record: 0 if <your_condition_here> else 1

logging_dict = {
    ...
    'filters': {
         'ignore_progress': {
            '()': my_filter_wrapper,
        }
    },
    ...

Or even simpler if your custom filtering logic is a one-liner and independent on the log record instance:

logging_dict = {
    ...
    'filters': {
         'ignore_progress': {
            '()': lambda : lambda _: 0 if <your_condition> else 1
        }
    },
    ...

It took me a long while to figure this out. Hope it helps anyone has the same questions.

And there is definitely something needed in its Python implementation to make it more elegant.

score 0 · Answer 2 · answered Sep 12 '19 at 10:13

0

I suggest using loguru as logging package. you can easily add a handler for your logger.

answered Sep 12 '19 at 10:13

mosi_kha

512
6
7

1

Thanks for your answer. But that's not what I need. I want to understand how python logger filter works. – user44875 Sep 12 '19 at 15:23

score 0 · Answer 3 · answered Oct 02 '20 at 01:11

This is not working because it is a bug or the docs are not correct.

In either case, I opened a ticket with the python folks here: https://bugs.python.org/issue41906

Workaround

If you return a function that returns a function things will work fine.

For example:

def no_error_logs():
    """
    :return: function that returns 0 if log should show, 1 if not
    """
    return lambda param: 1 if param.levelno < logging.ERROR else 0

Then

    "filters": {
        "myfilter": {
            "()": no_error_logs,
        }
    },

Note: Your filter function returns true/false.

According to the docs: Filter functions are answering the question:

Is the specified record to be logged? Returns zero for no, nonzero for yes.

So you would need to adjust your function accordingly.

How to use a callable as a filter in logging python

3 Answers3

Workaround

Linked