Avoid Duplicate URL Crawling

Question

I coded a simple crawler. In the settings.py file, by referring to scrapy documentation, I used

DUPEFILTER_CLASS = 'scrapy.dupefilter.RFPDupeFilter'

If I stop the crawler and restart the crawler again, it is scraping the duplicate urls again. Am I doing something wrong?

Can you tell me whether you had to import a corresponding package? I added the `DUPEFILTER_CLASS` to my settings.py file, but got the following error: `ModuleNotFoundError: No module named 'scrapy.dupefilter'` — Lena, May 29 '19 at 09:37

score 30 · Accepted Answer · edited May 28 '17 at 04:47

30

I believe what you are looking for is "persistence support", to pause and resume crawls.

To enable it you can do:

scrapy crawl somespider -s JOBDIR=crawls/somespider-1

You can read more about it here.

edited May 28 '17 at 04:47

Sebastián Palma

32,692
6
40
59

answered Jan 25 '14 at 00:07

Jason Youk

802
2
8
20

score 8 · Answer 2 · edited May 23 '17 at 12:34

According to the documentation, DUPEFILTER_CLASS is already set to scrapy.dupefilter.RFPDupeFilter by default.

RFPDupeFilter doesn't help if you stop the crawler - it only works while actual crawling, helps you to avoid scraping duplicate urls.

It looks like you need to create your own, custom filter based on RFPDupeFilter, like it was done here: how to filter duplicate requests based on url in scrapy. If you want your filter to work between scrapy crawl sessions, you should keep the list of crawled urls somewhere in the database, or csv file.

Hope that helps.

score 0 · Answer 3 · answered Oct 28 '16 at 15:06

0

you can rewrite Scheduler with Redis like scrapy-redis then you can avoid duplicate URL crawling when reruning your project.

answered Oct 28 '16 at 15:06

wyx

3,334
6
24
44

Avoid Duplicate URL Crawling

3 Answers3

Linked