How can I hold back (buffer) logging output and flush it later only when meeting some criteria

Question

I'm using the python logging module to output logs to the console. I have the problem that I'm iterating through a list of objects in my code and while doing so I'm generating some log messages. For each object I decide if I need to modify it or not. I only want to output all the log information for that object when it is modified but that information is not available by the time I generate the log message.

So what I would need is some sort of buffer that saves all log messages that are generated while I handle the object and at the end when I decide to modify the object I want the messages printed to console and if I don't modify the object I want the messages to get flushed without printing them to the console. Some pseudo code:

log = logging.getLogger(__name__)
log.setLevel(logging.INFO)
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
fmt = logging.Formatter("%(levelname)s: %(message)s")
ch.setFormatter(fmt)
log.addHandler(ch)

for o in objects:
    object_modified = False
    log.info('Analyzing {}'.format(o.name))
    ... do stuff ...
    if not o.is_active:
         log.warning('The object is inactive')
    log.info('Some generic message')
    if o.minimum < 0:
        o.minumum = 0
        object_modified = True
    if object_modified:
        # Magic here ...
        make_log_print_to_console()
    else:
        make_log_entries_disappear()

So all log calls I make in that for loop (whatever the level) should be held back right until the end where they either get all printed to the console or they just get flushed without any output depending on the object_modified variable.

Any idea how to do this (in an elegant way) with the logging module?

But .. it violates idea of logging. Fails have to be logged as well — Olvin Roght, Nov 23 '20 at 10:31
@OlvinRoght fails will raise an exception and can be handled outside of the loop. This is a special use case and I wouldn’t do that in general. — Sebastian Wiesinger, Nov 23 '20 at 10:38

score 0 · Answer 1 · answered Nov 23 '20 at 11:26

You can use a StringIO object to act as a buffer

In [1]: import logging

In [2]: from io import StringIO

In [3]: from contextlib import contextmanager

In [4]: @contextmanager
   ...: def log_buffer(log_handler):
   ...:     with StringIO() as buffer:
   ...:         original_stream = log_handler.stream
   ...:         log_handler.stream = buffer
   ...:         try:
   ...:             yield buffer
   ...:         finally:
   ...:             log_handler.stream = original_stream
   ...:

In [5]: log = logging.getLogger()
   ...: log.setLevel(logging.INFO)
   ...: ch = logging.StreamHandler()
   ...: ch.setLevel(logging.DEBUG)
   ...: fmt = logging.Formatter("%(levelname)s: %(message)s")
   ...: ch.setFormatter(fmt)
   ...: log.addHandler(ch)

In [6]: log.info('hello')
INFO: hello

In [7]: with log_buffer(ch) as buffer:
   ...:     log.info('not printed')
   ...:

In [8]: with log_buffer(ch) as buffer:
   ...:     log.info('printed')
   ...:     print(buffer.getvalue())
   ...:
INFO: printed

In [9]: for i in range(10):
   ...:     with log_buffer(ch) as buffer:
   ...:         log.info(f'my number if {i}')
   ...:         if i % 2 == 0:
   ...:             print(buffer.getvalue())
   ...:
INFO: my number if 0

INFO: my number if 2

INFO: my number if 4

INFO: my number if 6

INFO: my number if 8

Basically, you redirect the logger to a buffer, and you can choose to print the value of the buffer or to discard it. In the example, you always log the number but only print it when the number is even

Ps. in regard to using a new StringIO each iteration instead of clearing it, this answer says its better this way

How can I hold back (buffer) logging output and flush it later only when meeting some criteria

1 Answers1