5

(I have searched and not found a duplicate for this question but happy to be proved otherwise).

I need to rotate a log from within some Python code. The code is running on Windows (Server 2008 R2).

Initially I used TimedRotatingFileHandler (from Python's logging.handlers package) but this doesn't work as we need due to what I understand is an issue it has with multi-processing (subprocess.check_call is used to kick off another application).

I have checked out ConcurrentLogHandler which looks like it might do the job but I'm a bit concerned that it hasn't been updated since 2013 (although issues have been raised more recently).

UPDATE: an open bug (since 2013) indicates that ConcurrentLogHandler does not work with Python 2.7/Windows. On logging, the code just hangs.

Is there a best practice Windows solution I should be using?

azp74
  • 1,841
  • 2
  • 17
  • 23
  • It looks like the bug you are referring to is for a 3rd party package, not one that is included with Python – Mike Driscoll May 26 '16 at 13:40
  • 1
    I found a fellow who is using a logging server to get this to work: https://www.huyng.com/posts/python-logging-from-multiple-processes – Mike Driscoll May 26 '16 at 13:45
  • 1
    This might also be useful - http://plumberjack.blogspot.com/2010/09/using-logging-with-multiprocessing.html – Mike Driscoll May 26 '16 at 13:50
  • Yes - ConcurrentLogHandler has a bug but TimedRotatingFileHandler seems to have issues too (see https://bugs.python.org/issue4749 - very long discussion which references other issues). I am probably going to go ahead & use a queue to manage this (plumberjack). Frustrating as I'm not attempting to log from my second process. – azp74 May 31 '16 at 04:20

3 Answers3

0

Maybe I'm missing something, but Python's logging module comes with a RotatingFileHandler:

import logging
import time

from logging.handlers import RotatingFileHandler

#----------------------------------------------------------------------
def create_rotating_log(path):
    """
    Creates a rotating log
    """
    logger = logging.getLogger("Rotating Log")
    logger.setLevel(logging.INFO)

    # add a rotating handler
    handler = RotatingFileHandler(path, maxBytes=20,
                                  backupCount=5)
    logger.addHandler(handler)

    for i in range(10):
        logger.info("This is test log line %s" % i)
        time.sleep(1.5)

#----------------------------------------------------------------------
if __name__ == "__main__":
    log_file = r"c:\path\to\test.log"
    create_rotating_log(log_file)

This worked fine for me with Python 2.7 on Windows 7. Here are a couple of links that go into more detail:

Mike Driscoll
  • 32,629
  • 8
  • 45
  • 88
  • 1
    Yes - this (and TimedRotatingFileHandler, which is what I've used as the original reqt was for logs to roll over every day) works absolutely fine until you throw multiple processes into the mix. A similar issue was logged as a bug (subsequently closed as not a bug) in 2008 (https://bugs.python.org/issue4749). In my case my extra process is not attempting to write to the log but we see the same behaviour. I will update my question to clarify this point! – azp74 May 26 '16 at 00:19
0

OK - this is what I ended up doing.

Because as far as the logging is concerned things are only multithreaded (additional process does not write to my log) I have ended up hand cranking this. I'm not thrilled by this approach - create a threading lock object, close the logging (logging.shutdown - not thrilled about that either, as the doco says to call it on program exit ...), move the file & start up logging again. Then release the lock.

It is all in a try/except block so that if something goes wrong the lock is released.

Testing indicates that this does what is required.

Does calling logging.shutdown in this kind of context have some repercussions I'm not aware of?!

azp74
  • 1,841
  • 2
  • 17
  • 23
0

QueueHandler, which is addressed in the comment of the original question, is available from Python 3.2.

Python documentation also suggests using SocketHandler to send all logs into a socket server that dedicates the file writing process.

J. Choi
  • 1,616
  • 12
  • 23