4

I am trying to download a .csv from the following website: https://www.nasdaq.com/market-activity/stocks/screener?exchange=nyse&letter=0&render=download

Any help would be appreciated.

A5omic
  • 65
  • 8
  • 1
    The main problem is that the URL is not right. You need to have a URL of an actual file, not the download page. On the website you want to do it, it's not possible to get the URL of the file – Billy Bonaros Jan 27 '21 at 04:44
  • 1
    I took a look at the site and it gets the data from `https://api.nasdaq.com/api/screener/stocks`. Interstingly enough it replies with json and the javascript on the page must convert it to csv fro you to download. – Arthur Borshenko Jan 27 '21 at 04:52
  • What information do you want exactly? We may do it using pandas data-reader! – Billy Bonaros Jan 27 '21 at 04:53
  • @BillyBonaros I just really want to grab all of the data that is in the table. So the symbol, name, last sale, net change, % change, and market cap. – A5omic Jan 27 '21 at 04:59
  • @Tom Have you tried just downloading the csv via the browser? then you can read it into pandas from with `pandas.read_csv` – Arthur Borshenko Jan 27 '21 at 05:01
  • @ArthurBorshenko are you saying to try something like this? https://stackoverflow.com/questions/35371043/use-python-requests-to-download-csv – A5omic Jan 27 '21 at 05:05
  • @Tom No that wouldn't work, because there isn't a csv file served directly by the webserver. I meant like going to the website yourself and clicking the download button – Arthur Borshenko Jan 27 '21 at 05:12
  • @ArthurBorshenko oh. My mistake, I mean I could do that however I was hoping to automate the data grabbing, and not having to do it manually – A5omic Jan 27 '21 at 05:24
  • 1
    Like what @ArthurBorshenko said - instead of trying to GET the CSV you can try making the API call to grab the JSON from their API endpoint instead. https://api.nasdaq.com/api/screener/stocks?tableonly=true&limit=3296&exchange=nyse - this URL should return all the data you need and you can convert the JSON into a dictionary and create the CSV. – Joe Akanesuvan Jan 27 '21 at 05:59
  • @JoeAkanesuvan So how would I go about doing that? Sorry, I am just not too sure what I need to do. I tried to follow something like this, however, it gets stuck in a loop trying to get the URL. https://stackoverflow.com/questions/65288379/how-to-convert-json-file-into-csv-using-python – A5omic Jan 28 '21 at 04:49
  • Impossible! How could you get 4 points vote up for that question? If I would have ask this question, then I would get -4 votes down. Haha – euraad Jan 26 '22 at 23:42

1 Answers1

4

As @JoeAkanesuvan has pointed out, the information is obtained via a JSON API. This can be accessed using the requests library. It can then be converted into a CSV file using Python as follows:

import requests
import csv

headers = {
    "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0",
}

url = "https://api.nasdaq.com/api/screener/stocks?tableonly=true&limit=3296&exchange=nyse"
r = requests.get(url, headers=headers)
j = r.json()

table = j['data']['table']
table_headers = table['headers']

with open('Stocks.csv', 'w', newline='') as f_output:
    csv_output = csv.DictWriter(f_output, fieldnames=table_headers.values(), extrasaction='ignore')
    csv_output.writeheader()

    for table_row in table['rows']:
        csv_row = {table_headers.get(key, None) : value for key, value in table_row.items()}
        csv_output.writerow(csv_row)

I would recommend you print(j) to better understand the structure of the data being returned.

This would give you output starting:

Symbol,Name,Last Sale,Net Change,% Change,Market Cap
BABA,Alibaba Group Holding Limited American Depositary Shares each representing eight Ordinary share,$260.25,-5.67,-2.132%,"704,141,925,150"
TSM,Taiwan Semiconductor Manufacturing Company Ltd.,$121.74,-4.91,-3.877%,"631,343,640,000"
JNJ,Johnson & Johnson Common Stock,$167.88,-2.60,-1.525%,"441,951,263,775"
Martin Evans
  • 45,791
  • 17
  • 81
  • 97