How to retry for 404 link not found in scrapy?

Asked Jan 07 '12 at 09:13

Active Jan 07 '12 at 12:03

Viewed 3,942 times

11

Some sites give 404 error temporarily. But I paste in on browser it works. How to tell scrapy to retry the 404 status code links for 5 times.

asked Jan 07 '12 at 09:13

nizam.sp

4,002
5
39
63

Maybe you are making too many requests using scrapy and sites just returninig 404? – adt Jan 07 '12 at 09:15
Even if I try 1 requests per second, it fails sometimes. Is there a way to say in scrapy to retry 404 status? – nizam.sp Jan 07 '12 at 09:17

1 Answers1

19

There are two Scrapy settings relevant to what you need:

RETRY_HTTP_CODES: you should override the default value in your project to include 404
RETRY_TIMES: just set it to 5

answered Jan 07 '12 at 12:03

Pablo Hoffman

1,540
13
19