222

How do I use dictConfig? How should I specify its input config dictionary?

Mateen Ulhaq
  • 24,552
  • 19
  • 101
  • 135
David Wolever
  • 148,955
  • 89
  • 346
  • 502

6 Answers6

311

How about here! The corresponding documentation reference is configuration-dictionary-schema.

LOGGING_CONFIG = { 
    'version': 1,
    'disable_existing_loggers': True,
    'formatters': { 
        'standard': { 
            'format': '%(asctime)s [%(levelname)s] %(name)s: %(message)s'
        },
    },
    'handlers': { 
        'default': { 
            'level': 'INFO',
            'formatter': 'standard',
            'class': 'logging.StreamHandler',
            'stream': 'ext://sys.stdout',  # Default is stderr
        },
    },
    'loggers': { 
        '': {  # root logger
            'handlers': ['default'],
            'level': 'WARNING',
            'propagate': False
        },
        'my.packg': { 
            'handlers': ['default'],
            'level': 'INFO',
            'propagate': False
        },
        '__main__': {  # if __name__ == '__main__'
            'handlers': ['default'],
            'level': 'DEBUG',
            'propagate': False
        },
    } 
}

Usage:

import logging.config

# Run once at startup:
logging.config.dictConfig(LOGGING_CONFIG)

# Include in each module:
log = logging.getLogger(__name__)
log.debug("Logging is configured.")

In case you see too many logs from third-party packages, be sure to run this config using logging.config.dictConfig(LOGGING_CONFIG) before the third-party packages are imported.

To add additional custom info to each log message using a logging filter, consider this answer.

pdaawr
  • 436
  • 7
  • 16
Dave
  • 11,499
  • 5
  • 34
  • 46
  • 15
    There's an alternative place for specifying the `root` logger: at the top level of the dictionary. It is described in the [docs](https://docs.python.org/2/library/logging.config.html#logging-config-dictschema), has preference over the `['loggers']['']` when both are present, but in my opinion, `['loggers']['']` is more logical. See also discussion [here](http://stackoverflow.com/questions/20258986/root-logger-in-dictconfig) – Antony Hatchkins Sep 18 '14 at 05:41
  • 3
    All those concise, beautiful YAML snippets in the python logging.config docs just can't be read directly. Bummer. – JimB May 17 '18 at 18:19
  • Isn't this django-specific? What if I'm using a different framework (Flask, Bottle, etc), or not even working on a web application? – Adam Parkin Dec 18 '18 at 21:04
  • It feels like a cheat with `'disable_existing_loggers': False` as then you're maybe not configuring it whole cloth, but maybe reusing something that's already there.. If you set it to `True` then I don't seem to get any output. – Nick T Apr 12 '19 at 23:19
  • Hi @Dave, how can i use a custom class on `format` from `formatters`? – Rafa Acioly May 08 '19 at 13:22
  • Why are you recommending `ext://sys.stdout` ? shouldn't this be `sys.stderr`? – Shiplu Mokaddim Mar 06 '20 at 15:34
  • @ShipluMokaddim Most log messages are informational and therefore they are logged to stdout. Typically, stderr is reserved for errors printed when the application exits due to the error. Anyhow, you can customize it as you like. – Asclepius Apr 20 '20 at 18:26
  • 1
    How about the recoverable errors? Errors should not be in the stdout. It's only for program output. – Shiplu Mokaddim May 19 '20 at 13:42
  • @ShipluMokaddim Recoverable errors (typically without traceback) is exactly what the the `ERROR` log level is for. I also use the `WARNING` log level. Unrecoverable errors will exit the application. The `main()` function of the application in `__main__.py` can catch all unrecoverable errors and log them at the `EXCEPTION` log level before reraising the exception. – Asclepius May 27 '20 at 19:28
  • If you don't need to define logger in each module. You can just change root logger name like logging.root.name = 'YourLoggerName' and inside your module use standard root logger which is configured above like empty logger with your custom logger overridden name – Evgeniy Sobolev Aug 15 '20 at 08:13
74

The accepted answer is nice! But what if one could begin with something less complex? The logging module is very powerful thing and the documentation is kind of a little bit overwhelming especially for novice. But for the beginning you don't need to configure formatters and handlers. You can add it when you figure out what you want.

For example:

import logging.config

DEFAULT_LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'loggers': {
        '': {
            'level': 'INFO',
        },
        'another.module': {
            'level': 'DEBUG',
        },
    }
}

logging.config.dictConfig(DEFAULT_LOGGING)

logging.info('Hello, log')
pdoherty926
  • 9,895
  • 4
  • 37
  • 68
theotheo
  • 2,664
  • 1
  • 23
  • 21
  • 2
    This is the more relevant / useful example, at least in my case. It was the final `logging.info('Hello, log')` that made things click for me. The confusion in the documentation is that with dictConfig we no longer need to perform `getLogger` or any of those actions. – Mike Williamson Jun 19 '18 at 17:11
  • 1
    @theotheo Can you explain the empty key `'': { 'level': 'INFO'...` and why it doesnt work without it (e.g. when changing the blank value to a valid value such as `standard` – user9074332 Oct 01 '18 at 16:10
  • 1
    @MikeWilliamson: It can be useful, though, to still call `getLogger()` if you want multiple loggers with different names. Each of these loggers inherits configuration from the root logger. – Elias Strehle Nov 12 '18 at 07:40
  • 4
    @MikeWilliamson `getLogger` is always optional. When using the `logging.info()` method directly the root logger is used, while with `getLogger()` you can have different loggers, with differents names and levels. – sox supports the mods Mar 05 '19 at 10:54
41

Example with Stream Handler, File Handler, Rotating File Handler and SMTP Handler 

from logging.config import dictConfig

LOGGING_CONFIG = {
    'version': 1,
    'loggers': {
        '': {  # root logger
            'level': 'NOTSET',
            'handlers': ['debug_console_handler', 'info_rotating_file_handler', 'error_file_handler', 'critical_mail_handler'],
        },
        'my.package': { 
            'level': 'WARNING',
            'propagate': False,
            'handlers': ['info_rotating_file_handler', 'error_file_handler' ],
        },
    },
    'handlers': {
        'debug_console_handler': {
            'level': 'DEBUG',
            'formatter': 'info',
            'class': 'logging.StreamHandler',
            'stream': 'ext://sys.stdout',
        },
        'info_rotating_file_handler': {
            'level': 'INFO',
            'formatter': 'info',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': 'info.log',
            'mode': 'a',
            'maxBytes': 1048576,
            'backupCount': 10
        },
        'error_file_handler': {
            'level': 'WARNING',
            'formatter': 'error',
            'class': 'logging.FileHandler',
            'filename': 'error.log',
            'mode': 'a',
        },
        'critical_mail_handler': {
            'level': 'CRITICAL',
            'formatter': 'error',
            'class': 'logging.handlers.SMTPHandler',
            'mailhost' : 'localhost',
            'fromaddr': 'monitoring@domain.com',
            'toaddrs': ['dev@domain.com', 'qa@domain.com'],
            'subject': 'Critical error with application name'
        }
    },
    'formatters': {
        'info': {
            'format': '%(asctime)s-%(levelname)s-%(name)s::%(module)s|%(lineno)s:: %(message)s'
        },
        'error': {
            'format': '%(asctime)s-%(levelname)s-%(name)s-%(process)d::%(module)s|%(lineno)s:: %(message)s'
        },
    },

}

dictConfig(LOGGING_CONFIG)
Andreas Haferburg
  • 5,189
  • 3
  • 37
  • 63
Yogesh Yadav
  • 4,557
  • 6
  • 34
  • 40
14

There's an updated example of declaring a logging.config.dictConfig() dictionary schema buried in the logging cookbook examples. Scroll up from that cookbook link to see a use of dictConfig().

Here's an example use case for logging to both stdout and a "logs" subdirectory using a StreamHandler and RotatingFileHandler with customized format and datefmt.

  1. Imports modules and establish a cross-platform absolute path to the "logs" subdirectory

    from os.path import abspath, dirname, join
import logging
from logging.config import dictConfig
base_dir = abspath(dirname(__file__))
logs_target = join(base_dir + "\logs", "python_logs.log")

  2. Establish the schema according to the dictionary schema documentation.

    logging_schema = {
    # Always 1. Schema versioning may be added in a future release of logging
    "version": 1,
    # "Name of formatter" : {Formatter Config Dict}
    "formatters": {
        # Formatter Name
        "standard": {
            # class is always "logging.Formatter"
            "class": "logging.Formatter",
            # Optional: logging output format
            "format": "%(asctime)s\t%(levelname)s\t%(filename)s\t%(message)s",
            # Optional: asctime format
            "datefmt": "%d %b %y %H:%M:%S"
        }
    },
    # Handlers use the formatter names declared above
    "handlers": {
        # Name of handler
        "console": {
            # The class of logger. A mixture of logging.config.dictConfig() and
            # logger class-specific keyword arguments (kwargs) are passed in here. 
            "class": "logging.StreamHandler",
            # This is the formatter name declared above
            "formatter": "standard",
            "level": "INFO",
            # The default is stderr
            "stream": "ext://sys.stdout"
        },
        # Same as the StreamHandler example above, but with different
        # handler-specific kwargs.
        "file": {  
            "class": "logging.handlers.RotatingFileHandler",
            "formatter": "standard",
            "level": "INFO",
            "filename": logs_target,
            "mode": "a",
            "encoding": "utf-8",
            "maxBytes": 500000,
            "backupCount": 4
        }
    },
    # Loggers use the handler names declared above
    "loggers" : {
        "__main__": {  # if __name__ == "__main__"
            # Use a list even if one handler is used
            "handlers": ["console", "file"],
            "level": "INFO",
            "propagate": False
        }
    },
    # Just a standalone kwarg for the root logger
    "root" : {
        "level": "INFO",
        "handlers": ["file"]
    }
}

  3. Configure logging with the dictionary schema

    dictConfig(logging_schema)

  4. Try some test cases to see if everything is working properly

    if __name__ == "__main__":
    logging.info("testing an info log entry")
    logging.warning("testing a warning log entry")

[EDIT to answer @baxx's question]

  1. To reuse this setting across your code base, instantiate a logger in the script you call dictConfig() and then import that logger elsewhere

     # my_module/config/my_config.py
 dictConfig(logging_schema)
 my_logger = getLogger(__name__)

Then in another script 

    from my_module.config.my_config import my_logger as logger
    logger.info("Hello world!")
Brent
  • 1,195
  • 10
  • 9
  • 1
    how would this work if one wanted to use the schema across multiple modules? Here it's declared in the module in which it's used I think ? – baxx Aug 02 '21 at 15:11
  • I may suggest minor conciseness edit: `from my_module.config.my_config import log` and `log = getLogger(__name__)` – Anton Jan 03 '23 at 02:44
  • Maybe I misunderstood this, but at which point the `__main__` logger will be used? I use your extension to provide the logger across my codebase. The logger is created at package base in my `__init__.py` – MaKaNu May 25 '23 at 15:11
7

I found Django v1.11.15 default config below, hope it helps

DEFAULT_LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse',
        },
        'require_debug_true': {
            '()': 'django.utils.log.RequireDebugTrue',
        },
    },
    'formatters': {
        'django.server': {
            '()': 'django.utils.log.ServerFormatter',
            'format': '[%(server_time)s] %(message)s',
        }
    },
    'handlers': {
        'console': {
            'level': 'INFO',
            'filters': ['require_debug_true'],
            'class': 'logging.StreamHandler',
        },
        'django.server': {
            'level': 'INFO',
            'class': 'logging.StreamHandler',
            'formatter': 'django.server',
        },
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler'
        }
    },
    'loggers': {
        'django': {
            'handlers': ['console', 'mail_admins'],
            'level': 'INFO',
        },
        'django.server': {
            'handlers': ['django.server'],
            'level': 'INFO',
            'propagate': False,
        },
    }
}
夜一林风
  • 1,247
  • 1
  • 13
  • 24
2

One more thing in case it's useful to start from the existing logger's config, the current config dictionary is can be obtained via

import logging
logger = logging.getLogger()
current_config = logger.__dict__  # <-- yes, it's just the dict

print(current_config)

It'll be something like:

{'filters': [], 'name': 'root', 'level': 30, 'parent': None, 'propagate': True, 'handlers': [], 'disabled': False, '_cache': {}}

Then, if you just do

new_config=current_config

new_config['version']=1
new_config['name']='fubar'
new_config['level']=20
#  ...and whatever other changes you wish

logging.config.dictConfig(new_config)

You will then find:

print(logger.__dict__)

is what you'd hope for

{'filters': [], 'name': 'fubar', 'level': 20, 'parent': None, 'propagate': True, 'handlers': [], 'disabled': False, '_cache': {}, 'version': 1}
Radrich
  • 61
  • 4