1

My GUI has an "Update Database" button and every time the user presses it, I want to start a Scrapy spider that stores the data scraped into a Sqlite3 Database. I implemented qt5reactor, as this answer suggests, but now I'm getting a ReactorNotRestartable error when I press the update button for a second time. How can I get around this? I tried switching from CrawlerRunner to CrawlerProcess, but it still throws the same error (but maybe I'm doing it wrong, though). I also cannot use this answer, because q.get() locks the event loop, so the GUI freezes when I run the spider. I'm new to multiprocessing, so sorry if I'm missing something incredibly obvious.

In main.py

... # PyQt5 imports
import qt5reactor
from scrapy import crawler
from twisted.internet import reactor
from currency_scraper.currency_scraper.spiders.investor import InvestorSpider

class MyGUI(QMainWindow):

    def __init__(self):
        self.update_db_button.clicked.connect(self.on_clicked_update)
        ...

    def on_clicked_update(self):
        """Gives command to run scraper and fetch data from the website"""
        runner = crawler.CrawlerRunner(
            {
                "USER_AGENT": "currency scraper",
                "SCRAPY_SETTINGS_MODULE": "currency_scraper.currency_scraper.settings",
                "ITEM_PIPELINES": {
                    "currency_scraper.currency_scraper.pipelines.Sqlite3Pipeline": 300,
                }
            }
        )
        deferred = runner.crawl(InvestorSpider)
        deferred.addBoth(lambda _: reactor.stop())
        reactor.run() # has to be run here or the crawling doesn't start
        update_notification()

    ... # other stuff

if __name__ == "__main__":
   open_window()
   qt5reactor.install()
   reactor.run()

Error log:

Traceback (most recent call last):
  File "c:/Users/Familia/Documents/ProgramaþÒo/Python/Projetos/Currency_converter/main.py", line 330, in on_clicked_update
    reactor.run()
  File "c:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\venv\lib\site-packages\twisted\internet\base.py", line 1282, in run
    self.startRunning(installSignalHandlers=installSignalHandlers)
  File "c:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\venv\lib\site-packages\twisted\internet\base.py", line 1262, in startRunning
    ReactorBase.startRunning(self)
  File "c:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\venv\lib\site-packages\twisted\internet\base.py", line 765, in startRunning    
    raise error.ReactorNotRestartable()
twisted.internet.error.ReactorNotRestartable
雷笑东
  • 49
  • 1
  • 10

0 Answers0