2

I am writing a script that connects to N hosts via SSH ... queries the 3rd party system and extracts data and then displays all the collected data in a certain format.

I want to log all the actions the script is executing as well as any exceptions encountered on to the console and to a log file, so that the user can see what is happening while the script is running (If someone used Ansible - then just like the output we get on the console and logs when running the playbooks)

Expected output

  • [timestamp]: connecting machine 1
  • [timestamp]: connection established
  • [timestamp]: querying database xyz
  • [timestamp]: ERR: invalid credentials
  • [timestamp]: aborting data extraction
  • [timestamp]: connection closed
  • [timestamp]: ---------------------------
  • [timestamp]: connecting machine 2
  • [timestamp]: connection established
  • [timestamp]: querying database xyz
  • [timestamp]: extraction complete
  • [timestamp]: closing the connection

I hope I am able to explain it correctly - Logging actions and exceptions with timestamp for the whole script and all the data iterations.

Please advice and if possible with an example script that uses the technique. Thanks

slackmart
  • 4,754
  • 3
  • 25
  • 39
Buggy B
  • 623
  • 7
  • 18

2 Answers2

3

You can have a look here for some more detailed guidance. Here's how I usually set up logging on my stuff:

import logging

...

logger = logging.getLogger()
log_handler = logging.StreamHandler(sys.stdout)
log_handler.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s - %(funcName)s - line %(lineno)d"))
log_handler.setLevel(logging.DEBUG)
logger.addHandler(log_handler)
logger.setLevel(logging.DEBUG)

This will produce output like this, for any event from DEBUG upwards:

2017-05-16 13:30:03,193 - root - INFO - Starting execution - main - line 35
2017-05-16 13:30:03,206 - root - DEBUG - Config file grabbed successfully - readConfig - line 71
...
2017-05-15 13:30:26,792 - root - WARNING - Reached maximum number of attempts (3) for this request; skipping request. - main - line 79
2017-05-15 13:30:26,797 - root - ERROR - Failed to grab item. Unfortunately, this is a showstopper :( - main - line 79

The above is produced by a line in the main function of my app, that reads:

logger.info("Starting execution")

Another line in my readConfig function:

logging.debug("Config file grabbed successfully")

And another two lines in main again:

logging.warning("Reached maximum number of attempts ({max_attempts}) for this request; skipping request.".format(max_attempts = max_tries))
...
logging.error("Failed to grab item. Unfortunately, this is a showstopper :(")

Then it's a matter of how much information and context you need on each log entry. Have a look here at formatting the entries, and here at formatters. I'll have these sent to me via email, anytime the app runs triggered by crontab, by adding MAILTO = root to the top of my crontab file, and making sure my system email is properly set.

If you want to set it to go to the console and a file, you'll just need to set two different handlers. This answer provides a good example, where you'd set a StreamHandler to log to the console, and a FileHandler to log to a file. So instead of setting it up as I mentioned above I usually do, you could try:

import logging

...

# Set up logging and formatting
logger = logging.getLogger()
logFormatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s - %(funcName)s - line %(lineno)d")

# Set up the console handler
consoleHandler = logging.StreamHandler()
consoleHandler.setFormatter(logFormatter)
logger.addHandler(consoleHandler)

# Set up the file handler 
fileHandler = logging.FileHandler("{0}/{1}.log".format(logPath, fileName))
fileHandler.setFormatter(logFormatter)
logger.addHandler(fileHandler)

# Set up logging levels
consoleHandler.setLevel(logging.DEBUG)
fileHandler.setLevel(logging.DEBUG)
logger.setLevel(logging.DEBUG)
Graham
  • 7,431
  • 18
  • 59
  • 84
Marcy
  • 180
  • 1
  • 12
1

Check out the logging module here, there's a nice example section with both basic and advanced applications. Doing stuff in the format you've described appears to be included in the tutorial.

James Danylik
  • 371
  • 3
  • 6