How to refer to a standard library in a logging configuration file?

Question

I need to use a constant defined in the standard library socket in a logging configuration file. Problem, when reading the config file with logging.config.fileConfig() it ends with:

NameError: name 'socket' is not defined

My question is very close to this one, the difference is that if, as a workaround, I import the missing library (e.g. socket) from the main script reading this logging configuration file, it doesn't solve the problem (is this because I use python3?).

Complete logging configuration file:

[loggers]
keys=root,mainLogger

[handlers]
keys=mainHandler,nullHandler

[formatters]
keys=defaultFormatter,rawMessageFormatter

[logger_root]
level=INFO
handlers=nullHandler

[logger_mainLogger]
level=DEBUG
handlers=mainHandler
qualname=mainLogger

[handler_nullHandler]
class=NullHandler
args=(50,)

[handler_mainHandler]
class=logging.handlers.SysLogHandler
level=INFO
formatter=defaultFormatter
args=('/dev/log','myapp',socket.SOCK_STREAM)

[formatter_defaultFormatter]
format=%(asctime)s.%(msecs)d %(filename)s: %(funcName)s: %(message)s
datefmt=%Y/%m/%d %H:%M:%S

[formatter_rawMessageFormatter]
format=%(message)s
datefmt=

As another workaround I have tried the solution suggested here: How to use logging with python's fileConfig and configure the logfile filename but this neither works since socket.SOCK_STREAM is not a string (and I don't find any type that could work in the doc: https://docs.python.org/3.4/library/string.html#formatspec).

I have also tried to replace socket.SOCK_STREAM by 1 (since socket.SOCK_STREAM == 1 is True) but it doesn't work neither (socket.SOCK_STREAM not being an int...).

I would have liked to avoid converting my logging configuration file into a dictionary (but will do that if there's no other solution).

You mean, inside the logging configuration file? (this leads to a syntax error) — zezollo, Apr 21 '16 at 13:27
Strange, I have no syntax error using `1` or `__import__('socket').SOCK_STREAM`. What is your version of python ? — Cyrbil, Apr 21 '16 at 14:42

score 2 · Accepted Answer · answered Apr 21 '16 at 17:21

2

As documented in this section of the docs, the values are evaluated in the logging package's namespace. Hence, you can do something like this:

import logging
import socket

# The next line allows 'socket' in the logging package's namespace to pick up
# the stdlib socket module
logging.socket = socket
...
# when the config file is processed, it should work as expected
logging.config.fileConfig(...)
# remove the mapping from the logging package, as not needed any more
# (optional)
del logging.socket

answered Apr 21 '16 at 17:21

Vinay Sajip

95,872
14
179
191

This solves my problem without turning the config file into yaml, so it's better than the solution I found. Thanks! – zezollo Apr 21 '16 at 17:28
Just another comment to point out the differences between the two solutions: with your solution it's possible to use the configuration file I wrote in the question, as is. My solution forces to convert the configuration file into a yaml one, what must make use of `ext://socket.SOCK_STREAM` and not only `socket.SOCK_STREAM`; and the `logging.socket = socket` has no effect on it. – zezollo Apr 22 '16 at 04:15

score 0 · Answer 2 · answered Apr 21 '16 at 14:12

First solution (should work but doesn't)

Well here there is a partial answer: https://docs.python.org/3.4/library/logging.config.html#access-to-external-objects

So I tried this:

args=('/dev/log','mathmaker','ext://socket.SOCK_STREAM')

But it does not work:

Traceback (most recent call last):
  File "/usr/lib/python3.4/logging/__init__.py", line 1878, in shutdown
    h.close()
  File "/usr/lib/python3.4/logging/handlers.py", line 857, in close
    self.socket.close()
AttributeError: 'SysLogHandler' object has no attribute 'socket'

It's like python expects the 'external' object to be an attribute of the class declared in the handler section (e.g. here: class=logging.handlers.SysLogHandler).

Second solution (but requires to turn the config file into yaml):

So, as the mechanism that seems dedicated to solve this problem does not work, I have tried with a configuration file written in yaml, and now it works. It requires to add a dependency (python-yaml or python3-yaml for ubuntu users...) and to load the configuration file as a dictionary:

with open(settings.logging_conf_file) as f:
    logging.config.dictConfig(yaml.load(f))

and this way, it works.

Here is the same configuration file turned into working yaml (and notice that: 1. the import of socket is not required in the main script, looks like python will by itself 'magically' deal with the import; and 2. apart from the fact that yaml is easier to read than the old plain text config file, it also allows to define the keywords in a more readable way):

version: 1

formatters:
    rawMessageFormatter:
        format: '%(message)s'
        datefmt: ''
    defaultFormatter:
        format: '%(asctime)s.%(msecs)d %(filename)s: %(funcName)s: %(message)s'
        datefmt: '%Y/%m/%d %H:%M:%S'

handlers:
    nullHandler:
        class: logging.NullHandler
    mainHandler:
        class: logging.handlers.SysLogHandler
        level: INFO
        formatter: defaultFormatter
        address: '/dev/log'
        facility: 'myapp'
        socktype: ext://socket.SOCK_DGRAM

loggers:
    root:
        level: INFO
        handlers: [nullHandler]
    mainLogger:
        level: DEBUG
        handlers: [mainHandler]
        qualname: mainLogger

How to refer to a standard library in a logging configuration file?

2 Answers2

First solution (should work but doesn't)

Second solution (but requires to turn the config file into yaml):

Linked