2

So I've written a spider that extracts certain desired links from a webpage and puts the URL, link text, and other information not necessarily contained in the <a> tag itself, into an item for each link.

How should I pass this item onto another spider which scrapes the URL provided in that item?

wrongusername
  • 18,564
  • 40
  • 130
  • 214

1 Answers1

6

This question has been asked many times. Below are some links on this site that answer your question. Some answer it directly ie passing items to another function but you may realise that you do not need to do it that way, so other methods are linked to show whats possible.

Using multiple spiders at in the project in Scrapy

Scrapy - parse a page to extract items - then follow and store item url contents

Scrapy: Follow link to get additional Item data?

Community
  • 1
  • 1
zoonosis
  • 799
  • 1
  • 11
  • 30
  • 3
    None of these actually uses 2 spiders. What if first one finds links to csvs, second one should parse them and I want to use CSVSpider for that? – Suor Feb 02 '15 at 16:58