1

I was trying to download this data from the website. https://www.nseindia.com/market-data/oi-spurts

enter image description here

How can scrape this using python?

yash
  • 27
  • 5
  • Please try to search the 'downloadCSV()' function in the document using [Ctrl+F]. Maybe inside some tag you might find the download link of the CSV. – Ali Sajjad Jul 02 '20 at 20:14

2 Answers2

1

The JavaScript function downloadCSV is part of gijgo.min.js. It invokes getCSV, which goes through the fields in the table and generates a CSV file on the fly.

Fortunately, you don't have to deal with CSVs or scraping anything from the page. To get the data you want, all you have to do is make an HTTP GET request to the same RESTful API that your browser makes a request to when visiting the page:

def main():

    import requests

    url = "https://www.nseindia.com/api/live-analysis-oi-spurts-underlyings"

    headers = {
        "user-agent": "Mozilla/5.0"
    }

    response = requests.get(url, headers=headers)
    response.raise_for_status()

    data = response.json()["data"]

    print(f"There are {len(data)} items in total.")

    print(f"The first item is:\n{data[0]}")
    
    return 0


if __name__ == "__main__":
    import sys
    sys.exit(main())

Output:

There are 143 items in total.
The first item is:
{'symbol': 'MOTHERSUMI', 'latestOI': 7182, 'prevOI': 4674, 'changeInOI': 2508, 'avgInOI': 53.66, 'volume': 12519, 'futValue': 53892.6066, 'optValue': 3788085280, 'total': 55585.0344, 'premValue': 1692.4278, 'underlyingValue': 104}
>>>
Paul M.
  • 10,481
  • 2
  • 9
  • 15
  • How did you reach the API ? How can I use this to download all other data? we need t know the API addresses, like how did you get "https://www.nseindia.com/api/live-analysis-oi-spurts-underlyings" link – yash Jul 03 '20 at 18:29
  • 1
    @yash Take a look at the answer I posted here: https://stackoverflow.com/questions/61049188/how-do-i-get-this-information-out-of-this-website/61051360#61051360 – Paul M. Jul 03 '20 at 19:06
0

One way to find the final download link to a certain file is to open the debugger of your web browser and to click on the download link while looking into the Networking tab of the debugger.

Normally you will see the request the javascript of the page called as same as the url, the content of the request and so on...

From here you just need to replicate what request was sent by the javascript.

eliottness
  • 136
  • 3