Scrapy crawl from script always blocks script execution after scraping

Question

I am following this guide http://doc.scrapy.org/en/0.16/topics/practices.html#run-scrapy-from-a-script to run scrapy from my script. Here is part of my script:

    crawler = Crawler(Settings(settings))
    crawler.configure()
    spider = crawler.spiders.create(spider_name)
    crawler.crawl(spider)
    crawler.start()
    log.start()
    reactor.run()
    print "It can't be printed out!"

It works at it should: visits pages, scrape needed info and stores output json where I told it(via FEED_URI). But when spider finishing his work(I can see it by number in output json) execution of my script wouldn't resume. Probably it isn't scrapy problem. And answer should somewhere in twisted's reactor. How could I release thread execution?

Not sure I got your comment right. What do you mean 'call the script'? It hangs right in reactor.run() and log writes me: 'INFO: Closing spider (finished)' so spider seems to be finished. — Eugene Nagorny, Feb 08 '13 at 22:15

score 28 · Accepted Answer · answered Feb 10 '13 at 20:59

28

You will need to stop the reactor when the spider finishes. You can accomplish this by listening for the spider_closed signal:

from twisted.internet import reactor

from scrapy import log, signals
from scrapy.crawler import Crawler
from scrapy.settings import Settings
from scrapy.xlib.pydispatch import dispatcher

from testspiders.spiders.followall import FollowAllSpider

def stop_reactor():
    reactor.stop()

dispatcher.connect(stop_reactor, signal=signals.spider_closed)
spider = FollowAllSpider(domain='scrapinghub.com')
crawler = Crawler(Settings())
crawler.configure()
crawler.crawl(spider)
crawler.start()
log.start()
log.msg('Running reactor...')
reactor.run()  # the script will block here until the spider is closed
log.msg('Reactor stopped.')

And the command line log output might look something like:

stav@maia:/srv/scrapy/testspiders$ ./api
2013-02-10 14:49:38-0600 [scrapy] INFO: Running reactor...
2013-02-10 14:49:47-0600 [followall] INFO: Closing spider (finished)
2013-02-10 14:49:47-0600 [followall] INFO: Dumping Scrapy stats:
    {'downloader/request_bytes': 23934,...}
2013-02-10 14:49:47-0600 [followall] INFO: Spider closed (finished)
2013-02-10 14:49:47-0600 [scrapy] INFO: Reactor stopped.
stav@maia:/srv/scrapy/testspiders$

answered Feb 10 '13 at 20:59

Steven Almeroth

7,758
2
50
57

2

It definitely should be described in documentations. Thanks. – Eugene Nagorny Feb 13 '13 at 11:59
I've submitted a pull request which describes how to stop the reactor to the scrapy documentation, it should be in soon :) – Sjaak Trekhaak Jun 26 '13 at 06:54
1

When running scrapy from script like that how do I pass arguments for scrapy? Like `-o output.json -t json` – Igor Medeiros Sep 25 '13 at 21:44
where should I put that script please? – William Kinaan Feb 09 '14 at 18:06
could you help me here please? http://stackoverflow.com/questions/25353650/scrapy-how-to-import-the-settings-to-override-it – Marco Dinatsoli Aug 17 '14 at 21:19
1

instead of using an additional `stop_reactor`, this works: `crawler.signals.connect(reactor.stop, signal=signals.spider_closed)` – arcolife Sep 05 '14 at 07:42
@coldspeed Yes, have a look at the [documentation](https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script) – Steven Almeroth Feb 22 '19 at 14:13

Igor Medeiros · Answer 2 · 2013-12-04T20:39:35.657

In scrapy 0.19.x you should do this:

from twisted.internet import reactor
from scrapy.crawler import Crawler
from scrapy import log, signals
from testspiders.spiders.followall import FollowAllSpider
from scrapy.utils.project import get_project_settings

spider = FollowAllSpider(domain='scrapinghub.com')
settings = get_project_settings()
crawler = Crawler(settings)
crawler.signals.connect(reactor.stop, signal=signals.spider_closed)
crawler.configure()
crawler.crawl(spider)
crawler.start()
log.start()
reactor.run() # the script will block here until the spider_closed signal was sent

Note these lines

settings = get_project_settings()
crawler = Crawler(settings)

Without it your spider won't use your settings and will not save the items. Took me a while to figure out why the example in documentation wasn't saving my items. I sent a pull request to fix the doc example.

One more way to do it is just call command directly from you script

from scrapy import cmdline
cmdline.execute("scrapy crawl followall".split())  #followall is the spider's name

Scrapy crawl from script always blocks script execution after scraping

2 Answers2

Linked