Scrapyd api get and exeption when I try to start spider

Question

I have a issue about scrapyd api.

I write simple spider, it gets domain url as a argument.

import scrapy

class QuotesSpider(scrapy.Spider):
    name = 'quotes'

    def __init__(self, domains=None):
        self.allowed_domains = [domains]
        self.start_urls = ['http://{}/'.format(domains)]

    def parse(self, response):
        # time.sleep(int(self.sleep))
        item = {}
        item['title'] = response.xpath('//head/title/text()').extract()
        yield item

It works perfect if I run it like

scrapy crawl quotes -a domains=quotes.toscrape.com

But when time comes to run it via scrapyd_api it goes wrong:

from scrapyd_api import ScrapydAPI

scrapyd = ScrapydAPI('http://localhost:6800')
scrapyd.schedule(project='pd', spider='quotes',  domains='http://quotes.toscrape.com/')

I get - builtins.TypeError: init() got an unexpected keyword argument '_job'

How can I start scrapy spiders via scrapyd api with args?

score 1 · Accepted Answer · answered Jun 10 '19 at 14:08

it is an answer.

According to this answer I was wrong with super method.

now my code looks like this:

class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    start_urls = []

    def __init__(self, *args, **kwargs):
        super(QuotesSpider, self).__init__(*args, **kwargs)
        self.allowed_domains = [kwargs.get('domains')]
        self.start_urls.append('http://{}/'.format(kwargs.get('domains')))

Scrapyd api get and exeption when I try to start spider

1 Answers1