ReactorNotRestartable error in while loop with scrapy

Question

I get twisted.internet.error.ReactorNotRestartable error when I execute following code:

from time import sleep
from scrapy import signals
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
from scrapy.xlib.pydispatch import dispatcher

result = None

def set_result(item):
    result = item

while True:
    process = CrawlerProcess(get_project_settings())
    dispatcher.connect(set_result, signals.item_scraped)

    process.crawl('my_spider')
    process.start()

    if result:
        break
    sleep(3)

For the first time it works, then I get error. I create process variable each time, so what's the problem?

Answer 1

By default, CrawlerProcess's .start() will stop the Twisted reactor it creates when all crawlers have finished.

You should call process.start(stop_after_crawl=False) if you create process in each iteration.

Another option is to handle the Twisted reactor yourself and use CrawlerRunner. The docs have an example on doing that.

Answer 2

I was able to solve this problem like this. process.start() should be called only once.

from time import sleep
from scrapy import signals
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
from scrapy.xlib.pydispatch import dispatcher

result = None

def set_result(item):
    result = item

while True:
    process = CrawlerProcess(get_project_settings())
    dispatcher.connect(set_result, signals.item_scraped)

    process.crawl('my_spider')

process.start()

Answer 3

For a particular process once you call reactor.run() or process.start() you cannot rerun those commands. The reason is the reactor cannot be restarted. The reactor will stop execution once the script completes the execution.

So the best option is to use different subprocesses if you need to run the reactor multiple times.

you can add the content of while loop to a function(say execute_crawling). Then you can simply run this using different subprocesses. For this python Process module can be used. Code is given below.

from multiprocessing import Process
def execute_crawling():
    process = CrawlerProcess(get_project_settings())#same way can be done for Crawlrunner
    dispatcher.connect(set_result, signals.item_scraped)
    process.crawl('my_spider')
    process.start()

if __name__ == '__main__':
for k in range(Number_of_times_you_want):
    p = Process(target=execute_crawling)
    p.start()
    p.join() # this blocks until the process terminates

Answer 4

Ref http://crawl.blog/scrapy-loop/

 import scrapy
 from scrapy.crawler import CrawlerProcess
 from scrapy.utils.project import get_project_settings     
 from twisted.internet import reactor
 from twisted.internet.task import deferLater

 def sleep(self, *args, seconds):
    """Non blocking sleep callback"""
    return deferLater(reactor, seconds, lambda: None)

 process = CrawlerProcess(get_project_settings())

 def _crawl(result, spider):
    deferred = process.crawl(spider)
    deferred.addCallback(lambda results: print('waiting 100 seconds before 
    restart...'))
    deferred.addCallback(sleep, seconds=100)
    deferred.addCallback(_crawl, spider)
    return deferred


_crawl(None, MySpider)
process.start()

Answer 5

I faced error ReactorNotRestartable on AWS lambda and after I came to this solution

By default, the asynchronous nature of scrapy is not going to work well with Cloud Functions, as we'd need a way to block on the crawl to prevent the function from returning early and the instance being killed before the process terminates.

Instead, we can use `

import scrapy
import scrapy.crawler as crawler
rom scrapy.spiders import CrawlSpider
import scrapydo

scrapydo.setup()

# your spider
class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ['http://quotes.toscrape.com/tag/humor/']

    def parse(self, response):
        for quote in response.css('div.quote'):
            print(quote.css('span.text::text').extract_first())

scrapydo.run_spider(QuotesSpider)

` to run your existing spider in a blocking fashion:

Answer 6

I was able to mitigate this problem using package crochet via this simple code based on Christian Aichinger's answer to the duplicate of this question Scrapy - Reactor not Restartable. The initialization of Spiders is done in the main thread whereas the particular crawling is done in different thread. I'm using Anaconda (Windows).

import time
import scrapy
from scrapy.crawler import CrawlerRunner
from crochet import setup

class MySpider(scrapy.Spider):
    name = "MySpider"
    allowed_domains = ['httpbin.org']
    start_urls = ['http://httpbin.org/ip']

    def parse(self, response):
        print(response.text)
        for i in range(1,6):
            time.sleep(1)
            print("Spider "+str(self.name)+" waited "+str(i)+" seconds.")

def run_spider(number):
    crawler = CrawlerRunner()
    crawler.crawl(MySpider,name=str(number))

setup()
for i in range(1,6):
    time.sleep(1)
    print("Initialization of Spider #"+str(i))
    run_spider(i)

Answer 7

I had a similar issue using Spyder. Running the file from the command line instead fixed it for me.

Spyder seems to work the first time but after that it doesn't. Maybe the reactor stays open and doesn't close?

Answer 8

I could advice you to run scrapers using subprocess module

from subprocess import Popen, PIPE

spider = Popen(["scrapy", "crawl", "spider_name", "-a", "argument=value"], stdout=PIPE)

spider.wait()

Answer 9

If you're trying to get a flask or django or fast-api service that is running into this. You've tried all the things people suggest about forking a new process to run the reactor-- none of it seems to work.

Stop what you're doing and go read this: https://github.com/notoriousno/scrapy-flask

Crochet is your best opportunity to get this working within gunicorn without writing your own crawler from scratch.

Answer 10

My way is multiprocessing use Process #create spider

class PricesSpider(scrapy.Spider):
      name = 'prices'
      allowed_domains = ['index.minfin.com.ua']
      start_urls = ['https://index.minfin.com.ua/ua/markets/fuel/tm/']

    def parse(self, response):
        pass

Than I create func which run my spider

#run spider

from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from twisted.internet import reactor

def parser():
    configure_logging({'LOG_FORMAT': '%(levelname)s: %(message)s'})
    runner = CrawlerRunner()
    d = runner.crawl(PricesSpider)
    d.addBoth(lambda _: reactor.stop())
    reactor.run()

Than I create new Python file, import here func 'parser' and create schedule for my spider

#create schedule for spider

import schedule
from  import parser
from multiprocessing import Process


def worker(pars):
    print('Worker starting')
    pr = Process(target=parser)
    pr.start()
    pr.join()


def main():
    schedule.every().day.at("15:00").do(worker, parser)
    # schedule.every().day.at("20:21").do(worker, parser)
    # schedule.every().day.at("20:23").do(worker, parser)
    # schedule.every(1).minutes.do(worker, parser)
    print('Spider working now')
    while True:
        schedule.run_pending()


if __name__ == '__main__':
    main()