How to loop through URLs based on variable, such as date?

Question

This is an example of a URL, I want to use to scrape. The URL changes based on the date:

https://betfair-data-supplier-prod.herokuapp.com/api/daily_racing_results?date=2020-01-01

I want to scrape data from the 1st of January to the 30th of June. How would I create a for loop to do this.

For example:

for day and month in URL:
    request = requests.get(f'https://betfair-data-supplier-prod.herokuapp.com/api/daily_racing_results?date=2020-{month}-{day})

you can use `pandas.date_range` – deadshot Jul 06 '20 at 05:19 — deadshot, Jul 06 '20 at 05:19

Shivam Jha · Accepted Answer · 2020-07-06T05:43:14.177

First set start date and end date. Then loop through them by increment by one day. then append the current date of iteration to the date parameter of your query:

import datetime
import requests

start_date = datetime.date(2019, 12, 31)
end_date = datetime.date(2020, 6, 30)

day_count = (end_date - start_date).days

date = start_date
for i in range(day_count):
    date += datetime.timedelta(days=1)
    request = requests.get('https://betfair-data-supplier-prod.herokuapp.com/api/daily_racing_results?date=' + str(date))

score 0 · Answer 2 · answered Jul 06 '20 at 05:59

You can compare dates so increment by one day from the beginning of the time period

Import datetime as dt
delta = dt.timedelta(days=1)
i = dt.date(2020, 1, 1)
ed = dt.date(2020, 6, 30)
while i<=ed:
    i += delta
    print(f'this/day/{i}')

produces

this/day/2020-01-01
this/day/2020-07-02
this/day/2020-07-03
...
this/day/2020-06-29
this/day/2020-06-30

How to loop through URLs based on variable, such as date?

2 Answers2