Automatically delete old Python log files

Question

I have a Python program that runs daily. I'm using the logging module with FileHandler to write logs to a file. I would like each run's logs to be in its own file with a timestamp. However, I want to delete old files (say > 3 months) to avoid filling the disk.

I've looked at the RotatingFileHandler and TimedRotatingFileHandler but I don't want a single run's logs to be split across multiple files, even if a single run were to take days. Is there a built-in method for that?

Answer 1

The logging module has a built in TimedRotatingFileHandler:

# import module
from logging.handlers import TimedRotatingFileHandler
from logging import Formatter

# get named logger
logger = logging.getLogger(__name__)

# create handler
handler = TimedRotatingFileHandler(filename='runtime.log', when='D', interval=1, backupCount=90, encoding='utf-8', delay=False)

# create formatter and add to handler
formatter = Formatter(fmt='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)

# add the handler to named logger
logger.addHandler(handler)

# set the logging level
logger.setLevel(logging.INFO)

# --------------------------------------

# log something
logger.info("test")

Old logs automatically get a timestamp appended.

Every day a new backup will be created.

If more than 91 (current+backups) files exist the oldest will be deleted.

Answer 2

import logging
import time
from logging.handlers import RotatingFileHandler

logFile = 'test-' + time.strftime("%Y%m%d-%H%M%S")+ '.log'

logger = logging.getLogger('my_logger')
handler = RotatingFileHandler(logFile, mode='a', maxBytes=50*1024*1024, 
                                 backupCount=5, encoding=None, delay=False)
logger.setLevel(logging.DEBUG)
logger.addHandler(handler)

for _ in range(10000):
    logger.debug("Hello, world!")

Answer 3

As suggest by @MartijnPieters in this question, you could easily extend the FileHandler class in order to handle your own deletion logic. For example, my class will hold only the last "backup_count" files.

import os
import re
import datetime
import logging 
from itertools import islice


class TimedPatternFileHandler(logging.FileHandler):
    """File handler that uses the current time fo the log filename,
    by formating the current datetime, according to filename_pattern, using
    the strftime function.

    If backup_count is non-zero, then older filenames that match the base
    filename are deleted to only leave the backup_count most recent copies,
    whenever opening a new log file with a different name.

    """

    def __init__(self, filename_pattern, mode, backup_count):
        self.filename_pattern = os.path.abspath(filename_pattern)
        self.backup_count = backup_count
        self.filename = datetime.datetime.now().strftime(self.filename_pattern)


        delete = islice(self._matching_files(), self.backup_count, None)
        for entry in delete:
            # print(entry.path)
            os.remove(entry.path)
        super().__init__(filename=self.filename, mode=mode)

    @property
    def filename(self):
        """Generate the 'current' filename to open"""
        # use the start of *this* interval, not the next
        return datetime.datetime.now().strftime(self.filename_pattern)

    @filename.setter
    def filename(self, _):
        pass

    def _matching_files(self):
        """Generate DirEntry entries that match the filename pattern.

        The files are ordered by their last modification time, most recent
        files first.

        """
        matches = []
        basename = os.path.basename(self.filename_pattern)
        pattern = re.compile(re.sub('%[a-zA-z]', '.*', basename))

        for entry in os.scandir(os.path.dirname(self.filename_pattern)):
            if not entry.is_file():
                continue
            entry_basename = os.path.basename(entry.path)
            if re.match(pattern, entry_basename):
                matches.append(entry)
        matches.sort(key=lambda e: e.stat().st_mtime, reverse=True)
        return iter(matches)


def create_timed_rotating_log(path):
    """"""
    logger = logging.getLogger("Rotating Log")
    logger.setLevel(logging.INFO)

    handler = TimedPatternFileHandler('{}_%H-%M-%S.log'.format(path), mode='a', backup_count=5)

    logger.addHandler(handler)
    logger.info("This is a test!")

Answer 4

Get the date/time. See this answer on how to get the timestamp. If the file is older than the current date by 3 months. Then delete it with

import os
os.remove("filename.extension")

save this file to py2exe, then just use any task scheduler to run this job at startup.

Windows: open the run command and enter shell:startup, then place your exe in here.

On OSX: The old way used to be to create a cron job, this doesn't work in many cases from my experience anymore but still work trying. The new recommended way by apple is CreatingLaunchdJobs. You can also refer to this topic for a more detailed explanation.