Python Logging - Configuring Rotating File Handler From Config file

Question

I have the below config file which I am using to configure logging in my python modules , the issue with this is that it does not append the timestamp to the file-name , right now the log file created is file_handler.log but I want it to be file_handler-timestamp.log

{
"version": 1,
"disable_existing_loggers": true,
"formatters": {
  "simple": {
    "format": "%(asctime)s %(levelname)s %(filename)s %(lineno)s %(message)s",
    "datefmt": "%Y-%m-%d %H:%M:%S"
  }
},
"handlers": {
  "file_handler": {
    "level": "INFO",
    "class": "logging.handlers.TimedRotatingFileHandler",
    "formatter": "simple",
    "filename": "error.log",
    "backupCount": 10,
    "interval" : 1,
    "when": "s",
    "encoding": "utf8"
  }
},
"loggers": { },
"root": {
  "handlers": [
    "file_handler"
  ],
  "level": "DEBUG"
}
}

Working code without config

import logging
import time
from logging.handlers import RotatingFileHandler
    
logger = logging.getLogger('log_1')
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s', '%Y-%m-%d %H:%M:%S')
handler = RotatingFileHandler('log' + time.strftime('_%Y%m%d-%H%M') + '.log', maxBytes=2000, backupCount=10)
handler.setFormatter(formatter)
logger.addHandler(handler)

Workaround code which updated the config dynamically when program is initiated

import logging.config
import json
import time

fp = open("sample_config.json")
config = json.load(fp)
fp.close()
config['handlers']['file_handler']['filename'] += time.strftime('_%Y%m%d-%H%M') + '.log'
logging.config.dictConfig(config)

score 1 · Accepted Answer · answered Oct 12 '20 at 11:57

1

timestamp of when? When the file is rotated or when it is created?

If you use TimedRotatingFileHandler, then when the file is rotated, the timestamp is appended to the filename.

If you want the timestamp appended to the log filename, you have to subclass RotatingFileHandler and modify the __init__ method. Then point the "class" item of "file_handler" of the config file to your new class.

It is easy to do, but it may not be what you need. Just let the timestamp be appended to the rotated log filename or if you need to limit the size too, read this answer

answered Oct 12 '20 at 11:57

dragon2fly

2,309
19
23

I want every time the module is run a new log file to be generated and the timestamp should be appended to the log file – Infinite Oct 12 '20 at 12:37
@SL' then, subclassing `RotatingFileHandler` is a way to go – dragon2fly Oct 12 '20 at 12:43
@dragon2flye - I am able to get the file rotated by using the updated config , edited in my original question now , I have basically added - "interval" : 1,"when": "s" but for the same initiated python module run this spawns multiple files after each second, I want to generate only a single file for each module run and append the time stamp. So basically log files that are being generated per run with the timestamp appended , I am able to get that by 'working code' added into my original question but need to do this via a config. – Infinite Oct 12 '20 at 15:42
@dragon2flye - I think this could be resolved if there was a way to pass the name of file in the config dynamically - 'log' + time.strftime('_%Y%m%d-%H%M') but not sure if can be done from the config – Infinite Oct 12 '20 at 15:55
@Infinite you are using `TimedRotatingFileHandler` in your config file but `RotatingFileHandler` in your 'working code'. Your config file is saying that it will rotate every 1 second, so multiple files are created. Just change it back to `RotatingFileHandler` and remove the `interval` and `when` parameters. – dragon2fly Oct 12 '20 at 22:55
@Infinite Even though you could use a template name in the config file, there must be a code to interpret that too. That means you have to subclass `RotatingFileHandler ` and modify the `__init__` method. But simple is the best, your workaround code is sufficient. – dragon2fly Oct 12 '20 at 23:02

Python Logging - Configuring Rotating File Handler From Config file

1 Answers1