Removing handlers from python's logging loggers

Question

I am playing with Python's logging system. I have noticed a strange behavior while removing handlers from a Logger object in a loop. Namely, my for loop removes all but one handler. Additional call to .removeHandler removes the last handler smoothly. No error messages are issued during the calls.

This is the test code:

import logging
import sys
logging.basicConfig()
dbg = logging.getLogger('dbg')
dbg.setLevel(logging.DEBUG)

testLogger = logging.getLogger('mylogger')
sh = logging.StreamHandler(sys.stdout)
fh = logging.FileHandler('mylogfile.log')
dbg.debug('before adding handlers: %d handlers'%len(testLogger.handlers))
testLogger.addHandler(fh)
testLogger.addHandler(sh)

dbg.debug('before removing. %d handlers: %s'%(len(testLogger.handlers), 
                                              str(testLogger.handlers)))
for h in testLogger.handlers:
    dbg.debug('removing handler %s'%str(h))
    testLogger.removeHandler(h)
    dbg.debug('%d more to go'%len(testLogger.handlers))

#HERE I EXPECT THAT NO HANDLER WILL REMAIN    
dbg.debug('after removing: %d handlers: %s'%(len(testLogger.handlers), 
                                              str(testLogger.handlers)))
if len(testLogger.handlers) > 0:
    #Why is this happening?
    testLogger.removeHandler(testLogger.handlers[0])
dbg.debug('after manually removing the last handler: %d handlers'%len(testLogger.handlers))

I expect that at the end of the loop no handlers will remain in the testLogger object, however the last call to .removeHandler apparently fails, as can be seen from the output below. Nevertheless additional call to this function removes the handler as expected. Here is the output:

DEBUG:dbg:before adding handlers: 0 handlers
DEBUG:dbg:before removing. 2 handlers: [<logging.FileHandler instance at 0x021263F0>, <logging.StreamHandler instance at 0x021262B0>]
DEBUG:dbg:removing handler <logging.FileHandler instance at 0x021263F0>
DEBUG:dbg:1 more to go
DEBUG:dbg:after removing: 1 handlers: [<logging.StreamHandler instance at 0x021262B0>]
DEBUG:dbg:after manually removing the last handler: 0 handlers

More interestingly, if I replace the original loop with the following one, the loop works as expected and no handlers remain in the testLogger object at the end of the loop. Here is the modified loop:

while len(testLogger.handlers) > 0:
    h = testLogger.handlers[0]
    dbg.debug('removing handler %s'%str(h))
    testLogger.removeHandler(h)
    dbg.debug('%d more to go'%len(testLogger.handlers))

What explains this behaviour? Is this a bug or am I missing something?

`for h in list(testLogger.handlers)` – sherpya Jul 19 '14 at 00:04 — sherpya, Jul 19 '14 at 00:04

score 151 · Accepted Answer · edited Sep 23 '22 at 16:29

151

This isn't logger-specific behaviour. Never mutate (insert/remove elements) the list you're currently iterating on. If you need, make a copy. In this case testLogger.handlers.clear() should do the trick.

edited Sep 23 '22 at 16:29

Moot

2,195
2
17
14

answered Sep 20 '11 at 11:48

Cat Plus Plus

125,936
27
200
224

3

Solution is not so obvious and nice since we need to take into account thread safety which is obviously compromised here (take a look at addHandler implementation - locks are used there to modify handlers list). – Pavlo May 08 '18 at 07:34
8

A safer alternative would be to modify the existing list: `testLogger.handlers.clear()` – zenofpython Nov 21 '19 at 19:02
1

@zenofpython I suppose `testLogger.handlers.clear()` calls the `clear` method of `list` type? Why is it safter than `testLogger.handlers = []`? – Zheng Liu May 18 '20 at 08:50
7

Redefining a list rather than modifying the original list is a problem whenever you use an alias somewhere that points to the original list. Suppose: `a = b = [1, 2, 3]` `a = []` Notice that `b` still points to `[1, 2, 3]`. In fairness, it is unlikely that you would ever use such an alias *in this case*, but it is good practice to maintain consistency so you don't get any unexpected surprises from cutting corners. – zenofpython May 18 '20 at 09:05

score 22 · Answer 2 · edited Jun 02 '20 at 14:55

22

instead of mutating undocumented .handler:

Option 1

logging.getLogger().removeHandler(logging.getLogger().handlers[0])

this way you remove exactly the preexisting handler object via offical api. Or to remove all handlers:

logger = logging.getLogger()
while logger.hasHandlers():
    logger.removeHandler(logger.handlers[0])

Option 2

logging.config.dictConfig(config={'level': logging.DEBUG, 'handlers': []}

Not only removes but prevents its creation. List root will have [] handlers

edited Jun 02 '20 at 14:55

Tom

7,269
1
42
69

answered Apr 27 '20 at 11:09

droid192

2,011
26
43

5

Option 1 does not work for non-root loggers: the method `hasHandlers()` also checks the handlers of the ancestor loggers, so if an ancestor logger has an handler then `logger.hasHandlers()` will always evaluate to `True` and therefore `logger.handlers[0]` will end up raising an `IndexError`. – Géry Ogam Nov 24 '20 at 01:25
1

Going off option 2, you can also just set your instantiated logger object's handlers attribute to an empty array, such as `logger = logging.getLogger(__name__)` and `logger.handlers = []` – deesolie Nov 15 '21 at 22:53

hobs · Answer 3 · 2018-09-05T16:16:46.227

18

If you don't want to delete them all (thanks for the tip @CatPlusPlus):

testLogger.handlers = [
    h for h in testLogger.handlers if not isinstance(h, logging.StreamHandler)]

edited Sep 05 '18 at 16:16

answered Mar 29 '16 at 18:08

hobs

18,473
10
83
106

Sorry,I found this didn't work.for isinstance can't used to judge handler's type – fat_cheng Jun 01 '17 at 03:05
2

@fat_cheng Why can't isinstance work for that purpose? I don't seem to have a problem using the suggestion in this answer myself. – AlanSE Oct 19 '17 at 13:00
I'm using python3 in Ubuntu 16.04. I use `if (type(h) == logging.StreamHandler)]` instead. – fat_cheng Oct 27 '17 at 01:33
@fat_cheng you should never do that because `type` doesn't check parent types (inheritance), `isinstance` does: https://stackoverflow.com/a/1549854/623735 – hobs Sep 05 '18 at 16:14
1

For personnal purposes i wanted to check that a handker worked without spamming slack channel afterwards. `handlers = logger.handlers; logger.error('Removing slack handler'); logger.handlers = [handler for handler in logger.handlers if handler != slack_handler]` – cgte Oct 03 '18 at 10:17

Rohit · Answer 4 · 2023-01-02T11:00:43.840

I would say if the intent for removing the logging handlers is to prevent sending logs to additional handlers (this is the case most of the times) then an alternative approach I follow is to set propagate to False for your current logger.

logger = logging.getLogger(__name__)
logger.propagate = False

That way your log messages will only go to the handlers you explicitly add and will not propagate to parent loggers.

Autiwa · Answer 5 · 2018-11-14T15:13:28.137

3

I've just found out that you can also do that within a logging .ini file, with the following block:

[logger_stpipe]
handlers=
propagate=1
qualname=stpipe

It basically deactivates all handlers for a given logger. But it's somewhat limited because you have to know the Logger's name in advance.

edited Nov 14 '18 at 15:13

answered Apr 16 '18 at 15:10

Autiwa

8,112
2
13
11

Removing handlers from python's logging loggers

5 Answers5

Option 1

Option 2

Linked