How can I automatically generate datetime strings for each month (between a certain time range) to scrape a website?

Question

So I have a small python script that will ask the user for a username then it will download their logs for a certain twitch.tv channel from overrustlelogs.net

At the moment I just have 2 lists, one for months one for the year because the log URL takes the format of

http://overrustlelogs.net/Channel%20chatlog/MonthName%20Year/user.txt

months = ['September','October','November','December','January','February','March','April','May','June','July','August','September','October','November','December','January','February','March','April','May','June','July','August']
years = ['2014','2014','2014','2014','2015','2015','2015','2015','2015','2015','2015','2015','2015','2015','2015','2015','2016','2016','2016','2016','2016','2016','2016','2016']

As you can see my way of using a for loop and taking the index of each loop is pretty stupid because I have to manually edit the list each month.

What's a better way to do this? I want the program to download the logs from September 2014 to the current Month and Year

so it looks like you want to know "How can I generate a list containing every month and year from t0 until t1 ?" where `t0 = sep/14` and `t1 = jun/16`... you should edit your title — dot.Py, Jun 20 '16 at 13:53
I think the question isn't really about URLs and logs but how to generate a sequence of months. Probably best to delete the rest of the question. — Peter Wood, Jun 20 '16 at 15:22
See these questions https://stackoverflow.com/questions/6576187/get-year-month-for-the-last-x-months https://stackoverflow.com/questions/993358/creating-a-range-of-dates-in-python — Peter Wood, Jun 20 '16 at 15:23

score 1 · Accepted Answer · answered Jun 20 '16 at 14:08

Here's a simple script that will help you.

It creates a list with days between certain dates using datetime.timedelta().
Then it format these days using only month name and year (%B-%Y)
And finally remove duplicates by using an if not in condition while looping through the calculated days between t0 and t1.

Code:

t0 = '21-06-2014'
t1 = '07-01-2016'

#formatting in the proper way to use timedelta()
start = datetime.datetime.strptime(t0, "%d-%m-%Y")
end = datetime.datetime.strptime(t1, "%d-%m-%Y")

#generating dates using list comprehension and timedelta()
date_generated = [start + datetime.timedelta(days=x) for x in range(0, (end-start).days)]

#removing duplicates using loop and 'if not in'
dates = []

for date in date_generated:
    s = str(date.strftime("%B-%Y"))
    if s not in dates:
        dates.append(s)

#printing dates    
for date in dates:
    print date

Output:

June-2014
July-2014
August-2014
September-2014
October-2014
November-2014
December-2014
January-2015
February-2015
March-2015
April-2015
May-2015
June-2015
July-2015
August-2015
September-2015
October-2015
November-2015
December-2015
January-2016

With the dates list generated, we may now create the urls using:

for date in dates:
    MonthName = date.split('-')[0]
    Year = date.split('-')[1]
    url = 'http://overrustlelogs.net/Channel%20chatlog/' + MonthName + '%20' + Year + '/user.txt'
    print url

Output:

http://overrustlelogs.net/Channel%20chatlog/June%202014/user.txt
http://overrustlelogs.net/Channel%20chatlog/July%202014/user.txt
http://overrustlelogs.net/Channel%20chatlog/August%202014/user.txt
http://overrustlelogs.net/Channel%20chatlog/September%202014/user.txt
http://overrustlelogs.net/Channel%20chatlog/October%202014/user.txt
http://overrustlelogs.net/Channel%20chatlog/November%202014/user.txt
http://overrustlelogs.net/Channel%20chatlog/December%202014/user.txt
http://overrustlelogs.net/Channel%20chatlog/January%202015/user.txt
http://overrustlelogs.net/Channel%20chatlog/February%202015/user.txt
http://overrustlelogs.net/Channel%20chatlog/March%202015/user.txt
http://overrustlelogs.net/Channel%20chatlog/April%202015/user.txt
http://overrustlelogs.net/Channel%20chatlog/May%202015/user.txt
http://overrustlelogs.net/Channel%20chatlog/June%202015/user.txt
http://overrustlelogs.net/Channel%20chatlog/July%202015/user.txt
http://overrustlelogs.net/Channel%20chatlog/August%202015/user.txt
http://overrustlelogs.net/Channel%20chatlog/September%202015/user.txt
http://overrustlelogs.net/Channel%20chatlog/October%202015/user.txt
http://overrustlelogs.net/Channel%20chatlog/November%202015/user.txt
http://overrustlelogs.net/Channel%20chatlog/December%202015/user.txt
http://overrustlelogs.net/Channel%20chatlog/January%202016/user.txt

score 0 · Answer 2 · answered Jun 20 '16 at 16:46

Apart from datetime module, you can also take a look at calendar module. However, it does not have a built-in to iterate through months.

import calendar
from collections import namedtuple

MonthWithYear = namedtuple('MonthWithYear', ['month', 'year'])

def months_between(since, until):
    def between_including(array, el1, el2):
        since = array.index(el1)
        until = array.index(el2)
        yield from array[since: until + 1]# *including* last element too
    months = calendar.month_name
    if since.year == until.year:
        for month in between_including(list(months), since.month, until.month):
            yield MonthWithYear(month, since.year)
    else:
        yield from months_between(since, MonthWithYear(months[-1], since.year))
        for year in range(since.year + 1, until.year):
            yield from months_between(MonthWithYear(months[1], year), MonthWithYear(months[-1], year))
        yield from months_between(MonthWithYear(months[1], until.year), until)

since = MonthWithYear('January', 2016)
until = MonthWithYear('August', 2019)


for month in months_between(since, until):
    print(month.month, month.year)

How can I automatically generate datetime strings for each month (between a certain time range) to scrape a website?

2 Answers2