Storing logger messages in a string

Question

I wanted to store all the intermediate log messages (warn, info, error) to a string in Python, and report those log messages to the console at the end of program.

I tried to follow the steps outlined in http://opensourcehacker.com/2011/02/23/temporarily-capturing-python-logging-output-to-a-string-buffer/ but was unsuccessful .

Could somebody tell me a short, clean way to do this?

This is what I've tried for now:

log = logging.getLogger('basic_logger')
log.setLevel(logging.DEBUG)
report = ""

memory_handler = logging.handlers.MemoryHandler(1024*20, logging.ERROR, report)
memory_handler.setLevel(logging.DEBUG)
log.addHandler(memory_handler)
    
log.info("hello world")
    
memory_handler.flush()
    
print "report:", report

Could you explain what is going wrong? – Navneet Aug 13 '15 at 22:50 — Navneet, Aug 13 '15 at 22:50
See answer here: http://stackoverflow.com/a/9534960/459745 – Hai Vu Aug 14 '15 at 00:44 — Hai Vu, Aug 14 '15 at 00:44

score 45 · Accepted Answer · edited Mar 06 '18 at 10:09

It can be as simple as logging to a StringIO object:

import logging
try:
    from cStringIO import StringIO      # Python 2
except ImportError:
    from io import StringIO

log_stream = StringIO()    
logging.basicConfig(stream=log_stream, level=logging.INFO)

logging.info('hello world')
logging.warning('be careful!')
logging.debug("you won't see this")
logging.error('you will see this')
logging.critical('critical is logged too!')

print(log_stream.getvalue())

Output

INFO:root:hello world
WARNING:root:be careful!
ERROR:root:you will see this
CRITICAL:root:critical is logged too!

If you want to log only those messages at levels WARN, INFO and ERROR you can do it with a filter. LevelFilter below checks each log record's level no, allowing only those records of the desired level(s):

import logging
try:
    from cStringIO import StringIO      # Python 2
except ImportError:
    from io import StringIO

class LevelFilter(logging.Filter):
    def __init__(self, levels):
        self.levels = levels

    def filter(self, record):
        return record.levelno in self.levels

log_stream = StringIO()    
logging.basicConfig(stream=log_stream, level=logging.NOTSET)
logging.getLogger().addFilter(LevelFilter((logging.INFO, logging.WARNING, logging.ERROR)))

logging.info('hello world')
logging.warning('be careful!')
logging.debug("you won't see this")
logging.error('you will see this')
logging.critical('critical is no longer logged!')

print(log_stream.getvalue())

Output

INFO:root:hello world
WARNING:root:be careful!
ERROR:root:you will see this

score 20 · Answer 2 · answered Oct 23 '20 at 06:22

Note that solutions involving basicConfig set attributes of the root logger which all other loggers inherit from, this can be unwanted because libraries will also log to it. My use case is a website that calls a data processing module, and I only want to capture that module's logs specifically. This also has the advantage of allowing existing handlers that log to file and the terminal to persist:

import io, logging
from django.http import HttpResponse

log_stream = io.StringIO()
log_handler = logging.StreamHandler(log_stream)
logging.getLogger('algorithm.user_output').addHandler(log_handler)

algorithm()
return HttpResponse(f'<pre>{log_stream.getvalue()}</pre>')

In algorithm.py:

logger = logging.getLogger(__name__ + '.user_output')  # 'algorithm.user_output'

This requires to use django, which is quite a large dependency... — Erel Segal-Halevi, Aug 15 '23 at 10:12
@ErelSegal-Halevi No, this just requires the built-in `logging` package. I was just using some code from my Django project as an example — xjcl, Aug 15 '23 at 11:46

score 6 · Answer 3 · answered Sep 10 '17 at 09:45

You can also write your own stream class. As https://docs.python.org/2/library/logging.handlers.html says, only writeand flushare used for the streaming.

Example:

import logging

class LogStream(object):
    def __init__(self):
        self.logs = ''

    def write(self, str):
        self.logs += str

    def flush(self):
        pass

    def __str__(self):
        return self.logs

log_stream = LogStream()
logging.basicConfig(stream=log_stream, level=logging.DEBUG)

log = logging.getLogger('test')
log.debug('debugging something')
log.info('informing user')

print(log_stream)

Outputs:

DEBUG:test:debugging something
INFO:test:informing user

A slightly better version would hold logs as a list, and append to it, with the `__str__` method returning `"".join(self.logs)`, but I like this version a bit more than the accepted answer (for one-off things, mind you) because you get more control over the behavior. — Anatoly Makarevich, May 03 '19 at 12:32

Federico Baù · Answer 4 · 2021-07-04T13:07:15.750

Quick Recipe to have multiple logger and use the StringIO as storage

Note:

This is an customized version of @mhawke Answer ---> HERE I needed to have multiple log going each one to do its things, here is a simple script that does that.

from io import StringIO
from datetime import date
# Formatter
LOG_FORMAT = '| %(asctime)s | %(name)s-%(levelname)s:  %(message)s '
FORMATTER = logging.Formatter(LOG_FORMAT)

# ------- MAIN LOGGER
main_handler = logging.StreamHandler()
main_handler.setLevel(logging.WARNING)
main_handler.setFormatter(FORMATTER)

# ------- FILE LOGGER
file_handler = logging.FileHandler(f'log_{date.strftime(date.today(), "%Y-%m-%d")}.log')
file_handler.setLevel(logging.INFO)
file_handler.setFormatter(FORMATTER)

# ------- SECONDARY STREAMER (HOLDS ALL THE LOGS FOR RETRIEVE LATER) LOGGER
streamer = StringIO()
stream_handler = logging.StreamHandler(stream=streamer)
stream_handler.setFormatter(FORMATTER)

# Root Logger
logging.basicConfig(level=10, handlers=[main_handler, file_handler, stream_handler]) # Add handlers to Logger
_logger = logging.getLogger(__name__)

_logger.log(10, "DEBUG MESSAGE")
_logger.log(20, "INFO MESSAGE")
_logger.log(30, "WARNING MESSAGE")
_logger.log(40, "ERROR!")
_logger.log(50, "CRITICAL")
print('==='*15)
print('\nGetting All logs from StringIO')
print(streamer.getvalue())

Clearing The Logs from StringIO

In addition, I needed to clear the Data an start from 0 again. The easiest way and faster by performance is just create a new StringIO instance and attach it to the StreamHandler instance.

new_streamer = StringIO()  # Creating the new instance
stream_handler.setStream(new_streamer)  # here we assign it to the logger

_logger.info("New Message")
_logger.info("New Message")
_logger.info("New Message")
print(new_streamer.getvalue()) # New data

Another way is to 'clear' the Stream, but as per this other **StackOverflow Answer by @Chris Morgan is less performant.

# Python 3
streamer.truncate(0)
streamer.seek(0)

_logger.info("New Message")
_logger.info("New Message")
_logger.info("New Message")
print(streamer.getvalue())

# Python 2
streamer.truncate(0)

_logger.info("New Message")
_logger.info("New Message")
_logger.info("New Message")
print(streamer.getvalue())

Documentation

score -1 · Answer 5 · answered Aug 13 '15 at 23:09

Maybe this example code is enough.
In general, you should post your code so we can see what is going on.
You should also be looking at the actual Python documentation for the logging module while you are following any given tutorial.
https://docs.python.org/2/library/logging.html

The standard Python logging module can log to a file. When you are done logging, you can print the contents of that file to your shell output.

# Do some logging to a file
fname = 'mylog.log'
logging.basicConfig(filename=fname, level=logging.INFO)
logging.info('Started')
logging.info('Finished')

# Print the output
with open(fname, 'r') as f:
    print f.read() # You could also store f.read() to a string

I don't want to use a file and then print the output of that file . I was looking for a method that directly logs to a string using some sort of a handler? --perhaps a memory handler? https://docs.python.org/2/library/logging.handlers.html — trup, Aug 13 '15 at 23:18

score -1 · Answer 6 · answered Jul 20 '22 at 11:31

We can use StringIO object for both python2 and python3 like this:

Python 3 ::

import logging

from io import StringIO

log_stream = StringIO()    
logging.basicConfig(stream=log_stream, level=logging.INFO)

logging.info('this is info')
logging.warning('this is warning!')
logging.debug('this is debug')
logging.error('this is error')
logging.critical('oh ,this is critical!')

print(log_stream.getvalue())

Similarly in Python 2::

import logging
from cStringIO import StringIO      

log_stream = StringIO()    
logging.basicConfig(stream=log_stream, level=logging.INFO)

logging.info('this is info')
logging.warning('this is warning!')
logging.debug('this is debug')
logging.error('this is error')
logging.critical('oh ,this is critical!')

print(log_stream.getvalue())

Output ::

INFO:root:this is info
WARNING:root:this is warning!
ERROR:root:this is error
CRITICAL:root:oh ,this is critical!

Storing logger messages in a string

6 Answers6

Quick Recipe to have multiple logger and use the StringIO as storage

Clearing The Logs from StringIO

Documentation

Linked