Python: How do i log requests to multiple files

Question

I'm writing a web-crawler and having an issue, I would like to log each different level of message to its own log file.

I thought this would work but it only creates 'crawler-debug.log'

logging.basicConfig(level=logging.DEBUG, filename='crawler-debug.log')
logging.basicConfig(level=logging.WARN, filename='crawler-warn.log')
logging.basicConfig(level=logging.ERROR, filename='crawler-error.log')
logging.basicConfig(level=logging.INFO, filename='crawler-info.log')

Where am I going wrong here...

Also I would like a time-stamp at the beginning of each line of the log-file so I know when the request was sent.

score 0 · Answer 1 · edited May 23 '17 at 12:20

You will need to create a file handler for each log level. Formatting can be applied to each handler to add the timestamp. Python Logging Cookbook - Multiple handlers and formatters

Also to filter out other log levels other than the one you want (e.g. only DEBUG logging going to debug log file), you'll need to write your own handler object. See similar question answered here.

Here's an example with just debug and info logging.

import logging

# Define handler to omit logs that are not DEBUG level
class DebugFileHandler(logging.FileHandler):
    def __init__(self, filename, mode='a', encoding=None, delay=False):
        logging.FileHandler.__init__(self, filename, mode, encoding, delay)

    def emit(self, record):
        if not record.levelno == logging.DEBUG:
            return
        logging.FileHandler.emit(self, record)

# Define handler to omit logs that are not INFO level
class InfoFileHandler(logging.FileHandler):
    def __init__(self, filename, mode='a', encoding=None, delay=False):
        logging.FileHandler.__init__(self, filename, mode, encoding, delay)

    def emit(self, record):
        if not record.levelno == logging.INFO:
            return
        logging.FileHandler.emit(self, record)


logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)
# create file handlers
dh = DebugFileHandler('debug.log')
dh.setLevel(logging.DEBUG)
ih = InfoFileHandler('info.log')
ih.setLevel(logging.INFO)
# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(message)s')
dh.setFormatter(formatter)
ih.setFormatter(formatter)
# add the handlers to logger
logger.addHandler(ih)
logger.addHandler(dh)

# 'application' code
logger.debug('debug message')
logger.info('info message')

While this link might answer the question, could you summarize the relevant parts in case it gets changed, removed? Thanks. — Gábor Bakos, Jan 28 '15 at 16:45
Wow this is getting dense, this is almost as many lines of code as my crawler... I suppose i should create a separate module for logging requests thus separating the logging logic from the crawler logic. — Ricky Wilson, Jan 28 '15 at 17:30
When I'm done writing the class I will post it here so that others can use and extent it. — Ricky Wilson, Jan 28 '15 at 17:32

Python: How do i log requests to multiple files

1 Answers1