How to use python logging module to specify filename for log file?

Question

I am trying to create a new log file every hour using the python module but it does not seem to be working.

Sometimes it creates just mylog and sometimes it contains the date mylog.YYYYMMDD for the filename

I only want it to be mylog.YYYYMMDD. Where is the error ?

My code:

logging.basicConfig(
level=logging.INFO

)

root_logger = logging.getLogger('')
handler = logging.handlers.TimedRotatingFileHandler("mylog",when='midnight',interval=1,backupCount=10)
handler.suffix = "%Y-%m-%d"
root_logger.addHandler(handler)

did you try formatting? something like this: logFormatter = logging.Formatter('%Y-%m-%d%(message)') handler.setFormatter( logFormatter ) — Dave S, Mar 13 '17 at 07:38
Hi, I tried but it returned an empty file; no logs were written. — decemberrobot, Mar 13 '17 at 07:46
Right now it rotates between 2 files, mylog and mylog.2017-03-13 13:00 respectively. But what I want to achieve is mylog.2017-03-13 13:00 and then mylog.2017-03-13 14:00 and keep creating a new log file every hour. — decemberrobot, Mar 13 '17 at 07:49
You only call - handler = logging.handlers.TimedRotatingFileHandler("mylog",when='midnight',interval=1,backupCount=10). subsequent calls to your logger should produce your desired result. If you keep calling TimeRotatingRileHandler, the logging will reset to just the log file name — Dave S, Mar 13 '17 at 08:12
What do you mean ? I only call it once. Which is shown in my code isn't it ? — decemberrobot, Mar 13 '17 at 08:18

Dave S · Accepted Answer · 2017-03-13T08:35:19.137

Calllogging.handlers.TimedRotatingFileHandler("mylog",when='midnight',interval=1,backupCount=10) only once, at the beginning of your logging session.

If you call the above line each time, you will create a new instance of the TimedRotatingFileHandler class and a new logging stream will created. That will produce the "mylog" without your timestamp as the logging is reset as described in the Python documentation:

https://docs.python.org/2/library/logging.handlers.html

Below is a simple example that shows what I describe above:

def create_timed_rotating_log(path):
    """"""
    logger = logging.getLogger("Rotating Log")
    logger.setLevel(logging.INFO)

    handler = TimedRotatingFileHandler(path,
                                   when="m",
                                   interval=1,
                                   backupCount=5)
    logger.addHandler(handler)

    for i in range(6):
        logger.info("This is a test!")
        time.sleep(75)

score 2 · Answer 2 · edited May 23 '17 at 12:00

First, a note: it's generally unwise to adjust the .suffix field of a RotatingFileHandler as there are actually two fields that depend on the log file name suffix. One is a literal suffix represented as a strftime format directive, and the other is a regular expression that is used to choose old log files to remove. If you set one without setting the other, you can make your code fail to remove files (this is the most likely), or to remove the wrong files (much less likely). Fortunately the .suffix you set is the .suffix already set by using when='midnight'.

Next, the design of these rotating file handlers is not what you are expecting. Instead, there is a notion of a current file, and then some number of saved backup files. The current file has no suffix: it's just named mylog. This is always the case.

Every once in a while—at the rotation interval-and-time, or some time after it has passed if you are logging infrequently enough—the logging module will notice that, oh look at the time/date/whatever, it's time for a new mylog file. At this point it:

renames the existing mylog file, if there is one, to a backup name using the suffix it chose based on your when argument;
creates a new mylog (unsuffixed) file; and
removes any extra backup files, if there are now "too many".

In other words, at or sometime after midnight—this is your chosen "when"; the actual sequence of events occurs only upon logging a message—your existing handler notices that it's time for a new file. It looks for mylog.<some-suffix> file names where the suffix part matches:

r"^\d{4}-\d{2}-\d{2}$"

(this is hardcoded for midnight; there are other hardcoded regular expressions for other when values). If there are fewer than backupCount (10) such files now, the existing ones remain in place. Otherwise enough of the oldest¹ of those files are removed so that there are only 9 remaining. The existing mylog then becomes mylog.2017-03-13 or similar (depending on the date of course), and a new mylog-with-no-suffix is opened to be the current log file.

If you are getting files named mylog.2017-03-13 13:00 (including a space and hour and minute), that indicates that you have fussed with the .suffix field in a way not shown in your example code. The precise-to-a-minute format (for when == 'm' or when == 'M') is:

        self.suffix = "%Y-%m-%d_%H-%M"
        self.extMatch = r"^\d{4}-\d{2}-\d{2}_\d{2}-\d{2}$"

which has an underscore and a hyphen, not a space and a colon, at least in the source I am looking at.

You may also want to look at Using python logging from multiple modules with writing to a file and RotatingFileHandler for some general logging background.

¹This "oldest" depends on string sort order, which requires that the names of the files sort alphabetically the same way they would numerically. Hence the suffix must be year first, then month, then day, then hour, and so on: using month names like Jan, Feb, Apr would cause Apr to come before—i.e., be older than—Feb, which comes before Jan, and so on.

How to use python logging module to specify filename for log file?

2 Answers2