0

I am using the logging module in python inside a function. A simplified structure of the code is like below.

def testfunc(df):
    import logging
    import sys
    from datetime import datetime
    logger = logging.getLogger()
    logger.setLevel(logging.INFO)
    # to print to the screen
    ch = logging.StreamHandler(sys.__stdout__)
    ch.setLevel(logging.INFO) 
    logger.addHandler(ch)
    #to print to file
    fh = logging.FileHandler('./data/treatment/Treatment_log_'+str(datetime.today().strftime('%Y-%m-%d'))+'.log')
    fh.setLevel(logging.INFO) 
    logger.addHandler(fh)
    #several lines of code and some information like:
    logger.info('Loop starting...')

    for i in range(6):   # actually a long for-loop 
        #several lines of somewhat slow code (even with multiprocessing) and some information like:
        logger.info('test '+str(i))
            
    logging.shutdown()    

    return None

So, I know:

  • the logger need to be shutdown (logging.shutdown());
  • and it is included at the end of the function.

The issue is:

  • the actual function deals with subsets of a data frame, and sometimes it results in error because no sufficient data, etc.
  • If I run the function again, what I see is all messages are repeated twice (or even more, if I need to run again).
  • The situation remind the reported here, here, and here, for example... But slightly different...

I got, it is because the logging module was not shutdown, neither the handlers were removed... And I understand for the final function, I should anticipate such situations, and include steps to avoid raising errors, like shutdown the logger and finish the function, etc... But currently I am even using the log information to identify such situations...

My question is: how can I shut down it once such situation (function aborted because error) happened? ... in my current situation, in which I am just testing the code? Currently, the way to make it stop is to start the new console in Spyder (in my understanding, restarting the kernel). What is the correct procedure in this situation?

I appreciate any help...

hamagust
  • 728
  • 2
  • 10
  • 28
  • `logging.shutdown` is to be called at the end of an application. Once you call it, you can't reliably use loggers *anywhere* else in the application. – chepner Feb 24 '22 at 18:05
  • Dear @chepner, thank you for commenting so quickly. Yes, I understand... the (`logging.shutdown`) is at the end of that function... the point is... the error was thrown, and the function aborted... I am restarting the console/kernel... that's the only way? I need to have all previous data saved, loaded again, etc... – hamagust Feb 24 '22 at 18:12
  • Why are you calling `logging.shutdown()` in this function, if you plan to call the function again? Typically, you don't need to call `logging.shutdown` at all; it's registered with `atexit` when you import the module, so it gets called automatically at the end of the script. In what context are you using `testfunc`? – chepner Feb 24 '22 at 18:16
  • I do not intend to include the solution inside the function... at the end, I expect to circumvent all possible issues... Currently, I am using the log to track possible reasons for the errors in a file, but later I will just keep the log for control... my question is really in the current 'development' stage... – hamagust Feb 24 '22 at 18:18
  • The function is not intended to be called sequentially in a loop, neither so frequently... But when it is called, I would like to keep the procedures undertaken (how many records I have, when I got the data and treated, etc...)... I do not need the logs recorded in file from other parts (outside that function)... at least for now... it is not for an application, I am treating data for analysis... I imagined recording the log was the easier way to have a track on what was conducted specificlly for the procedure... – hamagust Feb 24 '22 at 18:24

1 Answers1

0

I suppose you can check first to see if there is any existing logger

loggers = [logging.getLogger(name) for name in logging.root.manager.loggerDict]

if there isn't, you can create the logger

if there is, don't create a new one

Alternatively, you can have another file setting the logger, and calling this file through an subprocess.Popen() or similar.

The code for the first option is from here How to list all existing loggers using python.logging module

eroc123
  • 624
  • 1
  • 4
  • 14