1

I have a problem with my Python script. When I just run my spider scrapy runspider Myspider it's work but if I run it from the main file I have this error : KeyError: 'driver'

Settings file :

SELENIUM_DRIVER_NAME = 'chrome'
#SELENIUM_DRIVER_EXECUTABLE_PATH = '/home/PATH/OF/FILE/chromedriver'
SELENIUM_DRIVER_ARGUMENTS=['--headless']

DOWNLOADER_MIDDLEWARES = {
    'scrapy_selenium.SeleniumMiddleware': 800
}

My spider file :

class MySpider(scrapy.Spider):
    name = 'my_spider'
    
    
    def __init__(self, list_urls, *args, **kwargs):
        super(my_spider, self).__init__(*args, **kwargs)
        self.urls = list_urls

    def start_requests(self):
        for url in self.urls:
            yield SeleniumRequest(
                url = url['link'],
                callback = self.parse,
                wait_time = 15,
            )

and my main file :

import scrapy
import classListUrls
from scrapy.crawler import CrawlerProcess
from dir.spiders import Spider


URL = "example.com"
urls = classListUrls.GenListUrls(URL)

process = CrawlerProcess()
process.crawl(Spider.my_spider, list_urls = urls.list_urls())
process.start()

I don't understand why this error.

SDLSpin
  • 13
  • 2

1 Answers1

0

One problem I see is, the first parameter to process.crawl should be the spider class, instead of the spider name.

process.crawl(Spider.MySpider, list_urls=urls.list_urls())

And the same is true when you call the superclass in the spiders __init__, although the better option would be to just leave it empty since the class is already the default.

class MySpider(scrapy.Spider):
    def __init__(self, *args, list_urls=None,**kwargs):
        super().__init__(*args, **kwargs)

Another thing is that the crawlerProcess needs to be constructed with a settings object because it doesn't read from the main settings.py file.

process = CrawlerProcess(settings={"SELENIUM_DRIVER_NAME": 'chrome',
                                   "SELENIUM_DRIVER_ARGUMENTS": ['--headless'],
                                   "DOWNLOADER_MIDDLEWARES": {'scrapy_selenium.SeleniumMiddleware': 800}})
Alexander
  • 16,091
  • 5
  • 13
  • 29
  • 1
    Great, I modified my script as you told me and everything works fine. Thanks a lot for your help, I was stuck for several days now. – SDLSpin Sep 18 '22 at 05:18
  • Hi Alexander, Can you help with this question? https://stackoverflow.com/q/73755333/8519380 – Sardar Sep 18 '22 at 16:28
  • 1
    @Sardar I have never used splash before... Ill take a look later tho – Alexander Sep 18 '22 at 17:22