0

I am trying to scrape the website in python, https://www.nseindia.com/
However when I try to load the website using Requests in python the call simply hangs below is the code I am using.

import requests  
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}  
r = requests.get('https://www.nseindia.com/',headers=headers)

The requests.get call simply hangs, not sure what I am doing wrong here? The same URL works perfectly in Chrome or any other browser.
Appreciate any help.

Merwyn Dsouza
  • 1
  • 1
  • I see you have a typo `headers=header` missing s at the end `headers=headers` also can you add what is the message at the end how mush time it waits for – Utpal Dutt Feb 11 '21 at 07:11
  • Similar problem found here: https://stackoverflow.com/questions/39227820/requests-process-hangs/39229526 – Colonel_Old Feb 11 '21 at 07:14
  • @UtpalDutt Corrected the code, it waits forever I waited 2 hrs befoer forcefully terminating the python process – Merwyn Dsouza Feb 11 '21 at 07:47
  • @MerwynDsouza alright! have you tried using selenium for this . I think the website is setup in a way they don't allow request like this moreover beautiful soup can fail to fetch the data if the data is dynamically generated I'll suggest using selenium. – Utpal Dutt Feb 11 '21 at 08:13
  • @MerwybDsouza, what data are you after? looks like you can get the data from direct XHR requests – chitown88 Feb 11 '21 at 09:02

0 Answers0