Using Scrapy to parse sitemaps

Question

I want to be able to use scrapy to crawl links on a sitemap. I don't know much about this application, so I would be interested in any links/info/documentation you could provide.

Thanks

score 12 · Accepted Answer · edited Feb 06 '12 at 13:08

12

A new generic spider has just been added to Scrapy trunk, for this purpose. It will be available on next release (Scrapy 0.14)

Code here: http://snippets.scrapy.org/snippets/20/
Documentation here: http://readthedocs.org/docs/scrapy/en/latest/topics/spiders.html#sitemapspider

edited Feb 06 '12 at 13:08

Bill the Lizard

398,270
210
566
880

answered Jun 15 '11 at 15:57

Pablo Hoffman

1,540
13
19

score 0 · Answer 2 · edited May 23 '17 at 12:02

0

All of the documentation is at http://doc.scrapy.org/. The tutorials can be found at scrapy.org also.

As for your question, see this SO question: how to parse a sitemap.xml file using scrapy's XmlFeedSpider?

edited May 23 '17 at 12:02

Community

1
1

answered Jun 13 '11 at 20:44

marr75

5,666
1
27
41

Using Scrapy to parse sitemaps

2 Answers2

Linked