Evaluate statements from within Python logging YAML config file

Question

Consider the following snippet of a Python logging YAML config file:

version: 1
formatters:
  simple:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
handlers:
  logfile:
    class: logging.handlers.TimedRotatingFileHandler
    level: DEBUG
    filename: some_fancy_import_name.generate_filename_called_error
    backupCount: 5
    formatter: simple

I would like to load this YAML config file this way:

with open('logging.yaml', 'r') as fd:
    config = yaml.safe_load(fd.read())
logging.config.dictConfig(config)

Take special notice of the filename to which the handler should write logs. In normal Python code, I would expect some_fancy_import_name.generate_filename_called_errorlog to generate the string 'error.log'. All in all, I would like to say that this logging handler should write to the file 'error.log' in the current directory.

However, as it turns out, this is not the case. When I look at the current directory, I see a file named 'some_fancy_import_name.generate_filename_called_errorlog'.

Why go through all this trouble?

I would like filename to be programmatically determined. I have successfully tried configuring logging using normal Python scripting this way:

# fancy import name
from os import environ as env

# Programmatically determine filename path
log_location = env.get('OPENSHIFT_LOG_DIR', '.')
log_filename = os.path.join(log_location, 'error')
handler = logging.handlers.TimedRotatingFileHandler(log_filename)

See how the log_filename path was inferred from environment variables.

I would like to translate this to a YAML config file. Is it possible?

Perhaps I might need to dig through the dict produced by yaml.safe_load(fd.read()) and do some eval() stuff?

Don't answer the question in the question itself. Either accept flyx's answer, or post your solution as an answer. — chepner, Jul 26 '16 at 13:27

score 5 · Accepted Answer · answered Jul 25 '16 at 17:50

You can add a custom constructor and mark the value with a special tag, so your constructor gets executed when loading it:

import yaml

def eval_constructor(loader, node):
  return eval(loader.construct_scalar(node))

yaml.add_constructor(u'!eval', eval_constructor)

some_value = '123'

config = yaml.load("""
version: 1
formatters:
  simple:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
handlers:
  logfile:
    class: logging.handlers.TimedRotatingFileHandler
    level: DEBUG
    filename: !eval some_value
    backupCount: 5
    formatter: simple
""")

print config['handlers']['logfile']['filename']

This prints 123, since the value some_value has the tag !eval, and therefore is loaded with eval_constructor.

Be aware of the security implications of evaling configuration data. Arbitrary Python code can be executed by writing it into the YAML file!

score 3 · Answer 2 · edited May 23 '17 at 12:01

Solution

Thanks to flyx's answer, this is how I did it:

import logging
import yaml
from os import environ as env

def constructor_logfilename(loader, node):
    value = loader.construct_scalar(node)
    return os.path.join(env.get('OPENSHIFT_LOG_DIR', '.'), value)

yaml.add_constructor(u'!logfilename', constructor_logfilename)
with open('logging.yaml', 'r') as fd:
    config = yaml.load(fd.read())
logging.config.dictConfig(config)

In the logging.yaml file, here's the important snippet:

...
filename: !logfilename error.log
...

score 0 · Answer 3 · answered Mar 16 '19 at 09:58

I just now had this issue come up myself and I approached it in a much more boring way. I left out the filename attribute of the logfile handler from my yaml file and added it to the config dictionary created by yaml.load like this:

with open('logging.yaml', 'r') as fd:
    config = yaml.load(fd.read())
config['handlers']['logfile']['filename'] = generate_logfile_name(...)
logging.config.dictConfig(config)

Evaluate statements from within Python logging YAML config file

Why go through all this trouble?

3 Answers3

Solution

Linked