spider_closed function is not fired

Question

I want to log some data when the spider finishes working.

I make this function inside my spider.

def spider_closed(self, spider):
        log.msg('The spider {0} finished working on'.format(self.name), level=log.INFO)

But this function is not getting fired.

I am using scrapy 0.20 with python 2.7

Please note that I can log to my log anything and actually I am logging a lot of data that I can see in my log file. so the logging is working with me.

Edit

I guess that I have to dispatch a signal signals.spider_closed but I don't know how to do that.

score 5 · Accepted Answer · edited May 23 '17 at 12:11

5

You should attach your method as a receiver to the SignalManager signal dispatcher:

from scrapy.xlib.pydispatch import dispatcher
from scrapy import signals

class MySpider(Spider):
    def __init__(self, *args, **kwargs):
        dispatcher.connect(self.spider_closed, signal=signals.spider_closed)
        super(MySpider, self).__init__(*args, **kwargs)

    def spider_closed(self, spider):
        log.msg('The spider {0} finished working on'.format(self.name), level=log.INFO)

Also see relevant threads:

edited May 23 '17 at 12:11

Community

1
1

answered Apr 21 '14 at 15:20

alecxe

462,703
120
1,088
1,195

I found the solution, it is almost the same like your answer, I did this: `dispatcher.connect(self.spider_closed, signals.spider_closed)` in my `__init__` and then, it works. is that correct please? – Simeoni Peraparota Apr 21 '14 at 15:22
@SimeoniPeraparota yup, it is nicer - edited the answer. Thanks. – alecxe Apr 21 '14 at 15:24
+1 to your appreciated efforts with me. will accept your answer when the system allows – Simeoni Peraparota Apr 21 '14 at 15:25

spider_closed function is not fired

I am using scrapy 0.20 with python 2.7

Edit

1 Answers1