Would Scrapy re request same URL if encountered in different domain?

Question

Imagine I have dont_filter = False and for current domain it will not send duplicate requests for already encountered URLs, but lets say I move to a different domain, will it send a new request?

Actually I just want to log URL's that have duplicate requests along with their referrer. What should be the appropriate place to capture these duplicate requests?

score 0 · Answer 1 · edited May 23 '17 at 12:07

0

Duplicate Requests are already being filtered and logged, set DUPEFILTER_DEBUG=True on settings to check those logs.

You could also create your own dupefilter implementation, check here.

There you could check the referers with on the Requests headers:

request.headers.get('referer')

edited May 23 '17 at 12:07

Community

1
1

answered May 02 '16 at 13:00

eLRuLL

18,488
9
73
99

I'm aware of duplicates being logged but where can I log(or use, sepertely) these URLs if I want to do something with those URLs? – Umar Iqbal May 02 '16 at 13:11
on your own dupefilter implementation. – eLRuLL May 02 '16 at 13:51

Would Scrapy re request same URL if encountered in different domain?

1 Answers1