Is there any way to change the log message format in scrapy?

Question

I would like to modify the scrapy log messages to contain user id at the beginning of it. for example, instead of this

2015-03-03 17:09:34+0530 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware

Is it possible to make it appear like

**user_id**:2015-03-03 17:09:34+0530 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware

Additionaly, is there a way to log only the user messages; i.e, Log only the messages I want logged which I will supply from the code?

Any help would be greatly appreciated. Thanks in advance!

score 3 · Answer 1 · answered Mar 21 '17 at 08:21

3

In settings.py add

LOG_FORMAT = user_id +' %(asctime)s %(levelname)8s\t%(message)s'

More info https://doc.scrapy.org/en/latest/topics/logging.html#logging-settings

answered Mar 21 '17 at 08:21

esamatti

18,293
11
75
82

score 2 · Accepted Answer · answered Apr 09 '15 at 13:38

Scrapy uses Twisted's logging systems which are build on top of logging standard library. I don't have answer for you, how to change date format of logs generated by scrapy itself.

For you second question I might have answer. I think you should completely silence scrapy logging by setting

LOG_ENABLED = False

in you settings.py. Then, configure logging module for yourself (on top of spider module)

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)8s\t%(message)s', )

and then log using logging.log, logging.info, logging.debug etc.

I leave to you figure out desired logging format.

Is there any way to change the log message format in scrapy?

2 Answers2