How to solve 403 error in scrapy

Question

I'm new to scrapy and I made the scrapy project to scrap data.

I'm trying to scrapy the data from the website but I'm getting following error logs

2016-08-29 14:07:57 [scrapy] INFO: Enabled item pipelines:
[]
2016-08-29 13:55:03 [scrapy] INFO: Spider opened
2016-08-29 13:55:03 [scrapy] INFO: Crawled 0 pages (at 0 pages/min),scraped 0 items (at 0 items/min)
2016-08-29 13:55:04 [scrapy] DEBUG: Crawled (403) <GET http://www.justdial.com/robots.txt> (referer: None)
2016-08-29 13:55:04 [scrapy] DEBUG: Crawled (403) <GET http://www.justdial.com/Mumbai/small-business> (referer: None)
2016-08-29 13:55:04 [scrapy] DEBUG: Ignoring response <403 http://www.justdial.com/Mumbai/small-business>: HTTP status code is not handled or not allowed
2016-08-29 13:55:04 [scrapy] INFO: Closing spider (finished)

I'm trying following command then on website console then I got the response but when I'm using same path inside python script then I got the error which I have described above.

Commands on web console:

$x('//div[@class="col-sm-5 col-xs-8 store-details sp-detail paddingR0"]/h4/span/a/text()')
$x('//div[@class="col-sm-5 col-xs-8 store-details sp-detail paddingR0"]/p[@class="contact-info"]/span/a/text()')

Please help me.

Thanks

Answer 1

Like Avihoo Mamka mentioned in the comment you need to provide some extra request headers to not get rejected by this website.

In this case it seems to just be the User-Agent header. By default scrapy identifies itself with user agent "Scrapy/{version}(+http://scrapy.org)". Some websites might reject this for one reason or another.

To avoid this just set headers parameter of your Request with a common user agent string:

headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0'}
yield Request(url, headers=headers)

You can find a huge list of user-agents here, though you should stick with popular web-browser ones like Firefox, Chrome etc. for the best results

You can implement it to work with your spiders start_urls too:

class MySpider(scrapy.Spider):
    name = "myspider"
    start_urls = (
        'http://scrapy.org',
    )

    def start_requests(self):
        headers= {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0'}
        for url in self.start_urls:
            yield Request(url, headers=headers)

Answer 2

Add the following script on your settings.py file. This works well if you are combining selenium with scrapy

headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0'}

Answer 3

I just needed to get my shell to work and run some quick tests so Granitosaurus's solution was a bit overkill for me.

I literally just went to the settings.py where you'll find mostly everything is commented out. In like line 16-17 or something you'll find something like this...

# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'exercise01part01 (+http://www.yourdomain.com)'

You just need uncomment it and replace it with any user agent like 'Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0'

You can find a list of them here https://www.useragentstring.com/pages/useragentstring.php[][1]

So it'll look something like this...

# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0'

You'll definitely want to rotate user agents if you want to make a make a large-scale crawler. But I just needed to get my scrapy shell to work and make some quick tests without getting that pesky 403 error so this one-liner sufficed. It was nice because I did not need to make a fancy function or anything.

Happy scrapy-ing

Note: PLEASE make sure you are in the same directory as settings.py when you run scrapy shell in order to utilize the changes you just made. It does not work if you are in a parent directory.

Answer 4

How could the whole process of error solution look like:

1. You can find a huge list of user-agents at https://www.useragentstring.com/pages/useragentstring.php, though you should stick with popular web-browser ones like Firefox, Chrome etc. for the best results (find more at How to solve 403 error in scrapy).

2. An example of steps working for me for Windows 10 in scrapy shell follows: https://www.useragentstring.com/pages/useragentstring.php -> choose 1 link from BROWSERS (but you also can try a link from links from CRAWLERS, ...) -> e.g. Chrome = https://www.useragentstring.com/pages/Chrome/ -> choose 1 of lines, e.g.: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36 -> choose 1 part (text that belongs together) from that line, e.g.: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) -> Command Prompt -> go into the project folder -> scrapy shell

    from scrapy import Request
    req = Request('https://www.whiskyshop.com/scotch-whisky?item_availability=In+Stock', headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6)'})
    fetch(req)

• Now, the result should be 200.

You see that it works even though I am on Windows 10 and there is Macintosh in Request().
You can use previous steps also to add a chosen header to file "settings.py".

Notes 1: Also comments in following stackoverflow web pages are more or less related (and I use them for this example):

https://stackoverflow.com/questions/52196040/scrapy-shell-and-scrapyrt-got-403-but-scrapy-crawl-works, 
https://stackoverflow.com/questions/16627227/problem-http-error-403-in-python-3-web-scraping, 
https://stackoverflow.com/questions/37010524/set-headers-for-scrapy-shell-request

Notes 2: I also recommend to read e.g.:

https://scrapeops.io/web-scraping-playbook/403-forbidden-error-web-scraping/
https://scrapeops.io/python-scrapy-playbook/scrapy-managing-user-agents/
https://www.simplified.guide/scrapy/change-user-agent