0

Is it possible to have the log level for [scrapy.core.engine] and [scrapy.extensions.logstats] to be 'INFO' as well as my custom logger and have everything else set to 'WARNING'? I'd like to do this to remove some of the clutter from my log file.

Thanks in advance!

EDIT:

I tried doing the following as described in this answer:

DEFAULT_LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'root': {'level': 'WARNING'},
    'loggers': {
        'twisted': {
            'level': 'ERROR',
        },
        'scrapy.core.engine': {
            'level': 'INFO',
        },
        'scrapy.extensions.logstats': {
            'level': 'INFO',
        },
    }
}

Unfortunately, this doesn't work for some reason. The default 'Warning' level doesn't get applied to all loggers and the specified levels for the loggers above seem to have no effect...

Max Smith
  • 925
  • 1
  • 14
  • 25

1 Answers1

1

Apparently scrapy uses logging. You could use something like the following:

import logging
for key in logging.Logger.manager.loggerDict:
    if "scrapy.core.engine" in key or "scrapy.extensions.logstat" in key:
        logging.getLogger(key).setLevel(logging.WARNING)
logging.basicConfig(level=logging.INFO)

I haven't tested it with scrapy, but I use it for other packages. Hope it works.

silgon
  • 6,890
  • 7
  • 46
  • 67
  • This definitely helps, however, I have not been able to implement it without messing up other loggers. When doing this the level for the other loggers goes to zero and I haven't been able to understand why yet. I have however explored a different approach noted in the edit to my question. Thank you. – Max Smith Aug 17 '17 at 22:40