10

I wanna know if it is possible to use multiple spiders within the same project together. Actually I need 2 spiders. The first one gathers the links on which the second spider should scrape. They both work on the same website, so the domain is similar.Is it possible? If yes can you give me an example? Thanks

Hossein
  • 40,161
  • 57
  • 141
  • 175

1 Answers1

14

Maybe this is what you're looking for: 

def parse(self, response):
    # parse the links (aka your first spider)
    for link in hxs('//XPATH'):
        yield Request(link.extract(), callback=self.parse_link)

def parse_link(self, response):
    # continue parsing (aka your second spider)

Hope this help you :)

anders
  • 825
  • 2
  • 10
  • 18
  • 4
    This doesn't actually answered the "multiple spiders" question. It's just a common hack that supplants the use of multiple spiders. Be good if there was more complete answer. – Zv_oDD Mar 07 '16 at 14:51