Downloading multiple XLS files from http by date in Python

Question

There's a general web address that can link to any XLS file inside the database. You cannot access it via ftp or directly through the directory but you can download from passing the date arguments into the http address itself.

The http is:

https://docs.misoenergy.org/marketreports/YYYYMMDD_sr_gfm.xls

A user can change the YYYYMMDD to an actual date and that address can be used to download the xls file (daily).

I want to be able to loop through, access these files and download them from a given start date and end date by substituting YYYY and MM and DD so start at 2003-01-01 and run til present time 2018-12-06.

How would I go about starting this?

[This](https://stackoverflow.com/questions/993358/creating-a-range-of-dates-in-python) could help — kstullich, Dec 06 '18 at 23:29

score 1 · Accepted Answer · answered Dec 06 '18 at 23:29

The following should work:

from datetime import date, timedelta

import requests

d1 = date(2018, 8, 15)  # start date
d2 = date(2018, 8, 25)  # end date

delta = d2 - d1

url_pattern = 'https://docs.misoenergy.org/marketreports/%s'
file_pattern = '%s_sr_gfm.xls'

for i in range(delta.days + 1):
    filename = file_pattern % str(d1 + timedelta(i)).replace('-', '')
    print(filename)

    response = requests.get(url_pattern % filename, stream=True)
    response.raise_for_status()

    with open(filename, 'wb') as handle:
        for block in response.iter_content(1024):
            handle.write(block)

Downloading multiple XLS files from http by date in Python

1 Answers1