using python logging to the the existing script

Question

I have a chain of python scripts which are working perfectly fine. What I am trying to do is to using python logging module to log all the print statements I have in this script (forget about the purpose of this script. All I am trying to do is to add logging capability to save all those print statements intended to show up on console to be logged in to a log file):

from __future__ import print_function
#import MySQLdb
import pymysql
import shutil
import os
import sys
import io
import string
import traceback
import time  
from watchdog.observers import Observer  
from watchdog.events import PatternMatchingEventHandler
from colorama import *
init()
# ==== FOR THE TRANSFORMATION SCRIPT ====
from datetime import tzinfo, timedelta, datetime
import pytz
import codecs
from pytz import timezone
import csv
# =======================================

if not os.path.exists('Archive'):
    os.mkdir("Archive")
if not os.path.exists('Failed'):
    os.mkdir("Failed")

class MyHandler(PatternMatchingEventHandler):
    patterns = ["*.csv","*.processing", "*.transforming","*.loading"]

    def process(self, event):
        """
        event.event_type 
            'modified' | 'created' | 'moved' | 'deleted'
        event.is_directory
            True | False
        event.src_path
            path/to/observed/file
        """
        eventFileName = event.src_path
        eventType = event.event_type
        if eventType == 'moved':
            eventFileName = event.dest_path
        fileNameWithPath, fileExtension = os.path.splitext(eventFileName)
        if fileExtension == '.csv':
            print (Back.GREEN +'Entering copier.py...'+ Style.RESET_ALL)
            execfile('./copier_MySQL.py')
        #if fileExtension == '.processing':
        #    print (Back.GREEN +'Entering parser_MySQL.py...'+ Style.RESET_ALL)
        #    execfile('./parser_MySQL.py')
        if fileExtension == '.processing': #change this to '.transforming' if the above parser_MySQL.py is uncommented
            t = time.time()
            print (Back.GREEN +'Entering transform_MySQL.py...'+ Style.RESET_ALL)
            execfile('./transform_MySQL.py')
            elapsed = time.time() - t
            print (Back.MAGENTA +'Time took to transform the file = '+str(elapsed)+ Style.RESET_ALL)
            print (Back.BLUE + "Exiting transform_MySQL.py..." + Style.RESET_ALL)
            print (Fore.YELLOW + "-----------#*#*#*-----------#*#*#*-----------" + Style.RESET_ALL)
        if fileExtension == '.loading':
            if os.path.isfile(eventFileName):
                t = time.time()
                print (Back.GREEN +'Entering sweeper.py...'+ Style.RESET_ALL)
                execfile('./sweeper_MySQL.py')
                elapsed = time.time() - t
                print (Back.MAGENTA +'Time took to load the file into database = '+str(elapsed-1)+ Style.RESET_ALL)
                print (Back.BLUE +"Disconnected from the MySQL server. Exiting sweeper.py..."+ Style.RESET_ALL)
                print (Fore.YELLOW+'-----------#*#*#*-----------#*#*#*-----------'+ Style.RESET_ALL)
            else:
                print (Fore.RED+eventFileName+' is not created by transform_MySQL.py. Check the above messages. NOT executing sweeper_MySQL.py'+ Style.RESET_ALL) 
    def on_moved(self, event):
        self.process(event)

    def on_created(self, event):
        self.process(event)

if __name__ == '__main__':
    observer = Observer()
    observer.schedule(MyHandler(), path='.')
    observer.start()

    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()

    observer.join()

I was looking at here and tried a few things but not able to quite fit that solution in my code. Any help is much appreciated.

Why not just redirect the output to a file? `myscript.py > myscript.log` — russdot, Mar 30 '16 at 19:25
@russdot this script is intended to run on un-monitored computer where the progress (those print statements) are to be printed to screen and save the log messages to a file, so `> myscript.log` is what I really wish I can use but can not. — Nikhil Gupta, Mar 30 '16 at 19:29
@russdot if you look at xi_'s comment to his answer below, he linked to python's logging module. All that functionality is automatically added to logging information buy a one line format statement. Doing this manually and writing to a log file is going to involve multiple lines of code which is kinda re-inventing the wheel, so is another reason for my reason to use python's logging module. — Nikhil Gupta, Mar 30 '16 at 20:50
I'm glad you found a solution that works. The post you linked to describes using the python `logging` module to log to a file (though not as simply as xi_'s answer), so I presumed you needed an alternative. — russdot, Mar 30 '16 at 21:01

xiº · Accepted Answer · 2016-03-30T19:28:19.480

3

You could use logging module instead of bunch of prints.

import logging
logging.basicConfig(level=logging.DEBUG)
logging.debug(Back.GREEN +'Entering sweeper.py...'+ Style.RESET_ALL)

To save output to file set:

logging.basicConfig(filename='myscript.log', level=logging.DEBUG)

edited Mar 30 '16 at 19:28

answered Mar 30 '16 at 19:25

xiº

4,605
3
28
39

that is what I thought but does it save those print statements to a file? – Nikhil Gupta Mar 30 '16 at 19:26
I will try it and get back. Also, can I use multiple levels in that statement something like `level=logging.DEBUG,logging.ERROR,logging.INFO` etc? – Nikhil Gupta Mar 30 '16 at 19:32
1

you could use messages with different level, for more info check this out https://docs.python.org/2/howto/logging.html – xiº Mar 30 '16 at 19:33
awesome, this worked and got to logging all including exceptions. :-) – Nikhil Gupta Mar 30 '16 at 20:46

using python logging to the the existing script

1 Answers1