Call a function everytime logging module in python is called

Question

I would like to call a function everytime, I do any sort of logging using the logging module in python.

As can be seen below, instead of calling copyfiles at each instance of logging, I would like to see if there is way to injest copyfiles to the filehandler in someway or even a wrapper.

from sys import argv
import shutil
import logging
logger = logging.getLogger('')

sh = logging.StreamHandler(sys.stdout)
logfile = 'logs/log-run-{}.log'.format(date.today().strftime('%d-%m-%Y'))
fh = logging.FileHandler(logfile)

formatter = logging.Formatter('[%(asctime)s] %(levelname)s \
                            [%(filename)s.%(funcName)s:%(lineno)d] \
                            %(message)s', datefmt='%a, %d %b %Y %H:%M:%S')
logger.setLevel(logging.INFO)

sh.setFormatter(formatter)
fh.setFormatter(formatter)

logger.addHandler(sh)
logger.addHandler(fh)

LOGPATH = args[1]
def copyfiles():
    """
    Copy files to azure blob storage
    """
    fh.flush()
    shutil.copy(logfile, LOGPATH)

logging.info('test')
copyfiles()
logging.info('foobar')
copyfiles()

I tried digging into invoking copyfiles each time logging is called, but I ended up no where.

In case you are wondering why I am copy files from logging, this is why.

So you want to copy the whole file every time a new line is added to it? — Julien Palard, Mar 16 '21 at 12:47
I have suggested a solution, let me know if that would work. Just curious, wouldn't this cause a ton of extra File I/O's and become a bottleneck ? — the23Effect, Mar 16 '21 at 14:38
This might become a bottleneck, but this is the only way out as stdout and stderr aren't given until the end of the job. so, i don't even know what i am waiting for — tandem, Mar 16 '21 at 14:54
I really think this is a very bad idea. Why not using another handler like an HTTPHandler, a SocketHandler, a DatagramHandler, whatever? — Julien Palard, Mar 16 '21 at 15:20
Do you have suggestions on how to incorporate it inside azure batch service? — tandem, Mar 16 '21 at 18:30
(I want to note that I would like to use this only in debug mode, and definetly NOT in production mode) — tandem, Mar 16 '21 at 18:37

score 2 · Accepted Answer · answered Mar 16 '21 at 14:34

Currently this is what I could think of:

Override the flush() of FileHandler by inheriting it into a utility class like FlushCopyFileHandler class shown in the code below.
Instead of using the FileHandler use the FlushCopyFileHandler class and all you have to do is call this overridden flush().

"""

from logging import FileHandler
from shutil

class FlushCopyFileHandler(FileHandler):
    
    # These arguments are passed along to the parent class FileHandler.
    def __init__(self, *args, **kwargs):
        super(FlushCopyFileHandler, self).__init__(filename, *args, **kwargs)
        self.copy_destination = "some default destination path"   # can also be set from args.
        
    # When this flush() is called it will call the flush() of FileHandler class.
    # After that it will call the shutil.copy() method.
    def flush(self):
        super().flush()
        shutil.copy(self.filename, self.copy_destination)
        
    # Use this to change destination path later on.
    def set_copy_path(self, destination):
        self.copy_destination = destination

"""

Call a function everytime logging module in python is called

1 Answers1