0

I have one spiner which scrape the page and gets all the urls.

i have another spiner which get a url and scrap on it.

i want to call the second spiner for each link i get from the first spiner.

the code for getting all links from the first spiner

for site in sites:
            Link = site.xpath('a/@href').extract()

but i don't know how to call the spiner for each Link

help please

Marco Dinatsoli
  • 10,322
  • 37
  • 139
  • 253

1 Answers1

1

I guess you better unite the two spiders and do something like:

def get_links(self, response):
    for site in sites:
        link = site.xpath('a/@href').extract()[0]
        yield Request(url=link, callback=self.scrape_them)

def scrape_them(self, response):
    # by now scrapy called the link and you get the response
    ...
Guy Gavriely
  • 11,228
  • 6
  • 27
  • 42