Python - Sort Log Messages By Level before Printing

Question

Let's say I have a logging setup like this in my Python script:

import logging

logging.basicConfig(level=logging.DEBUG, stream=sys.stdout,
                    format='%(asctime)s %(levelname)s %(message)s',
                    datefmt='%a, %d %b %Y %H:%M:%S')
logging.info('info')
logging.error('error...')
logging.debug('debug...')

Is there a way I can have it wait to print to stdout until the script is finished running and sort the log messages by level before printing?

score 3 · Accepted Answer · answered Oct 16 '12 at 17:29

3

From the looks of it, the object passed to stream only needs to have a write method. This means you can create a list-like object which appends data when write is called -- You can then sort the list-like object trivially before printing.

import logging

class LogList(list):
   def write(self,data):
       self.append(data)

LL = LogList()
logging.basicConfig(stream = LL,level=logging.DEBUG)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.error('Wow, this is bad')
logging.warning('And this, too')
logging.debug('foobar')
logging.warning('baz')

for line in sorted(LL):
   print line[:-1]

Of course, you might need to make your sort key a little more to pick up different levels. Something like:

levels = {'DEBUG':0,'INFO':1,'WARNING':2,'ERROR':3}
LL.sort(key = lambda x: levels[x.split(':')[0]])

answered Oct 16 '12 at 17:29

mgilson

300,191
65
633
696

+1 this is better than my method (I thought of something like this while writing up my answer, but mgilson beat me to it). It is more efficient, is less hack-y, and has much slicker sorting. – Matthew Adams Oct 16 '12 at 17:43
One thing I still don't like is that the logs are getting sorted by strings. Maybe we could use a [filter object](http://docs.python.org/library/logging.html#filter-objects) or something... – Matthew Adams Oct 16 '12 at 17:51
Wow this is so elegant! Thanks. What's the line[:-1] for? – Greg Oct 16 '12 at 18:26
I did "print line," so print doesn't add its own newline. – Greg Oct 16 '12 at 18:40
I implemented the filter object solution and posted it as another answer. – Matthew Adams Oct 16 '12 at 19:22

Matthew Adams · Answer 2 · 2012-10-16T17:38:36.920

It's pretty hack-y, but you could log to a StringIO object, split the lines, sort them, and then write the result to a file.

import logging
import cStringIO as StringIO

logStrObj = StringIO.StringIO()

logging.basicConfig(level=logging.DEBUG, stream=logStrObj,
                    format='%(asctime)s %(levelname)s %(message)s',
                    datefmt='%a, %d %b %Y %H:%M:%S')

logging.info('info')
logging.error('error...')
logging.info('info 2')
logging.debug('debug...')

# sort the contents of logStrObj
logList = logStrObj.getvalue().split('\n')
infoLogList = []
debugLogList = []
warningLogList = []
errorLogList = []
criticalLogList = []
for log in logList:
    if 'INFO' in log:
        infoLogList.append(log)
    elif 'DEBUG' in log:
        debugLogList.append(log)
    elif 'WARNING' in log:
        warningLogList.append(log)
    elif 'ERROR' in log:
        errorLogList.append(log)
    elif 'CRITICAL' in log:
        criticalLogList.append(log)
logList = infoLogList + debugLogList + warningLogList + errorLogList + criticalLogList

# write to a file (or print or whatever you want)
for line in logList:
    print line

Both my other answer and mgilson's answer are better than this, but I'll leave it here since it is a valid solution. — Matthew Adams, Oct 16 '12 at 19:23

score 1 · Answer 3 · edited May 23 '17 at 10:24

Here is a method that doesn't involve sorting after the fact (and so is a bit more efficient). It uses the single level filter from this question and separate handlers for each type of error. This way, the logs are definitely organized by type, and there won't be any problems from just checking strings to determine the log type.

import logging
import cStringIO as StringIO

class SingleLevelFilter(logging.Filter):
    '''This single level logging filter is from https://stackoverflow.com/a/1383365/1460235'''
    def __init__(self, passlevel, reject):
        self.passlevel = passlevel
        self.reject = reject

    def filter(self, record):
        if self.reject:
            return (record.levelno != self.passlevel)
        else:
            return (record.levelno == self.passlevel)

# Use this formatter and logLevel in place of setting the global ones
formatter = logging.Formatter(fmt='%(asctime)s %(levelname)s %(message)s',
                    datefmt='%a, %d %b %Y %H:%M:%S')
globalLogLevel = logging.INFO

# build handlers/ StringIO logs for each type
rootLogger = logging.getLogger()
rootLogger.setLevel(globalLogLevel)
logStrObjList = []
handlers = []
i = 0
for logLevel in [logging.INFO, logging.DEBUG, logging.WARNING, logging.ERROR, logging.CRITICAL]:
    logStrObjList.append(StringIO.StringIO())
    handlers.append(logging.StreamHandler(logStrObjList[i]))
    handlers[i].addFilter(SingleLevelFilter(logLevel, False))
    handlers[i].setFormatter(formatter)
    handlers[i].setLevel(globalLogLevel)
    rootLogger.addHandler(handlers[i])
    i += 1

logging.critical('bad news bears')
logging.info('info')
logging.error('error...')
logging.info('info 2')
logging.debug('debug...')
logging.error('another errooo')

Python - Sort Log Messages By Level before Printing

3 Answers3