Downloading csv file from a website using python

Question

I have the following website yahoo finance. I want to set a date range on that page for e.g April 3rd 1997 to 4th November 2015. Once I set the date range I get a link to download the file as csv on that page, below the tables. I want to download the csv file. But I want all of this to be done programmatically. How do I achieve this using python.

You didn't found anything using google? – Florin Ghita Nov 04 '15 at 07:10 — Florin Ghita, Nov 04 '15 at 07:10

H. U. · Answer 1 · 2015-11-13T22:54:39.960

This might be helpful :

import requests
import shutil
def callme():
    url = "http://real-chart.finance.yahoo.com/table.csv?s=%5EBSESN&a=03&b=3&c=1997&d=10&e=4&f=2015&g=d&ignore=.csv"
    r = requests.get(url, verify=False,stream=True)
    if r.status_code!=200:
        print "Failure!!"
        exit()
    else:
        r.raw.decode_content = True
        with open("file1.csv", 'wb') as f:
            shutil.copyfileobj(r.raw, f)
        print "Success"

if __name__ == '__main__':
    callme()

How to get this URL ?
You can get the list of API calls in any website by right click-> Inspect Element ->Network.
Now when you make any request from browser, it will list out all the API calls.

You can split the date according to your need and pass it in the URL. You need to do some research about how Yahoo passes the date in URL.

Edit 1: This script will run over HTTP and HTTPS.

score 3 · Answer 2 · edited Nov 04 '15 at 08:20

You can manipulate the URL to download the CSV you want, for example from April 3, 1997 to Nov 4, 2015 you can set the URL to

https://in.finance.yahoo.com/q/hp?s=%5EBSESN&a=[month]&b=[date]&c=[year]&d=[month]&e=[date]&f=[year]&g=[daily/weekly/monthly/dividends_only]

Month can be 00, 01, 02, 03, 04 up to 11 (Note: Month start from 00)

Date can start from 01 to 31

Year is in format 'yyyy'

For the daily -> d, for weekly -> w, for monthly -> m, for dividends only -> v

Now you have the URL, then to download the CSV you can just use link: URL=URL&ignore=.csv

import urllib
url = '[URL]'
csv = urllib.urlopen(url).read() # returns type 'str'
with open('file.csv', 'w') as fx: # str, hence mode 'w'
fx.write(csv)

Thanks, your answer helps, but when i am trying to open a url, which requires login acess, I got 'HTTP Error 403: Forbidden', how do i overcome this? — Noob Geek, Sep 16 '20 at 11:04

score 1 · Answer 3 · edited May 23 '17 at 12:17

1

You could try selenium. There are some other lightweight tools too, that can find by google.

Update: It seems you're looking for this.

edited May 23 '17 at 12:17

Community

1
1

answered Nov 04 '15 at 07:12

fireant

14,080
4
39
48

Isn't there a way to do this using a simple python script? I could construct the URL. Once I load the URL I get the html page. How do I download the csv file from the link on that page.? – liv2hak Nov 04 '15 at 07:19
see the update, using urllib2 you can do that – fireant Nov 04 '15 at 07:24

Downloading csv file from a website using python

3 Answers3

Linked