40

I want to download a bunch of files named with ISO-8601 dates. Is there a simple way to do this using bash+GNU coreutils? (Or some trick to make wget/curl to generate the list automatically, but I find that unlikely)

Similar to this question, but not restricted to weekdays: How to generate a range of nonweekend dates using tools available in bash?. I guess that there is a simpler way to do it without that restriction.

Also related to How to generate date range for random data on bash, but not restricted to a single year.

Community
  • 1
  • 1
Hjulle
  • 2,471
  • 1
  • 22
  • 34

5 Answers5

74

If you have GNU date, you could do use either a for loop in any POSIX-compliant shell:

# with "for"
for i in {1..5}; do 
    # ISO 8601 (e.g. 2020-02-20) using -I
    date -I -d "2014-06-28 +$i days"

    # custom format using +
    date +%Y/%m/%d -d "2014-06-28 +$i days"
done

or an until loop, this time using Bash's extended test [[:

# with "until"
d="2014-06-29"
until [[ $d > 2014-07-03 ]]; do 
    echo "$d"
    d=$(date -I -d "$d + 1 day")
done

Note that non-ancient versions of sh will also do lexicographical comparison if you change the condition to [ "$d" \> 2014-07-03 ].

Output from either of those loops:

2014-06-29
2014-06-30
2014-07-01
2014-07-02
2014-07-03

For a more portable way to do the same thing, you could use a Perl script:

use strict;
use warnings;
use Time::Piece;
use Time::Seconds;    
use File::Fetch;

my ($t, $end) = map { Time::Piece->strptime($_, "%Y-%m-%d") } @ARGV; 

while ($t <= $end) {
    my $url = "http://www.example.com/" . $t->strftime("%F") . ".log";
    my $ff = File::Fetch->new( uri => $url );
    my $where = $ff->fetch( to => '.' );  # download to current directory
    $t += ONE_DAY;
}

Time::Piece, Time::Seconds and File::Fetch are all core modules. Use it like perl wget.pl 2014-06-29 2014-07-03.

Tom Fenech
  • 72,334
  • 12
  • 107
  • 141
  • The first one can be combined with http://stackoverflow.com/a/8903280/939108 to use an explicit date. But I guess the second option is better then. – Hjulle Sep 06 '14 at 15:16
  • 1
    If you want a format other than ISO-8601, you can do `date -d $d +"%m/%d/"` or whatever formatting string strikes your fancy instead of the `echo $d` line above. – ijoseph Jun 17 '20 at 18:40
  • The `until` loop is better for "days in a month". – RonJohn Jan 18 '23 at 23:02
28

Using GNU date and bash:

start=2014-12-29
end=2015-01-03
while ! [[ $start > $end ]]; do
    echo $start
    start=$(date -d "$start + 1 day" +%F)
done

2014-12-29
2014-12-30
2014-12-31
2015-01-01
2015-01-02
2015-01-03
cweiske
  • 30,033
  • 14
  • 133
  • 194
glenn jackman
  • 238,783
  • 38
  • 220
  • 352
11

If you are on macOS, then date works a bit differently from GNU date. Here's a variant to Tom Fenech's date invocation that supports both GNU and Darwin:

if [ $(uname) = 'Darwin' ]; then
    d=$(date -j -v+1d -f %Y-%m-%d $d +%Y-%m-%d)
elif [ $(uname) = 'Linux' ]; then
    d=$(date -I -d "$d + 1 day")
fi
tink
  • 14,342
  • 4
  • 46
  • 50
Ernst de Haan
  • 467
  • 6
  • 12
  • 3
    Or just `brew install coreutils`, then use `gdate` – ijoseph Jun 17 '20 at 18:36
  • Check the day offset, needs a -v+${i}d for this to work in the @Tom Fenech example. MacOSX 12 zsh: ```start=2020-01-01; for i in {1..5}; do date -j -v+${i}d -f %Y-%m-%d "$start" +%Y-%m-%d; done ``` – Rich Andrews Mar 28 '22 at 18:26
2

I use this handy function to work with log files in the format yyyymmdd.log.gz: 

function datelist { for dt in $(seq -w $1 $2) ; do date -d $dt +'%Y%m%d' 2>/dev/null ; done ; }

It accepts dates in the format yyyymmdd.

runlevel0
  • 2,715
  • 2
  • 24
  • 31
  • 1
    This can of course be extremely inefficient if you need big date range. It will generate up to 70 useless days per month and 88 useless months per year, which it will feed to `date` which will give an error. – mivk Oct 08 '21 at 14:11
  • @mivk: Indeed. Good if you want a couple of years and if you run it on a fat machine. I haven't given it a second thought since I posted it. In my work, back then, I queried the SQL server, which can return a list of dates and does not need checking. But not everybody has an SQL server running and not all servers are as fast as the corporate one I used. – runlevel0 Oct 27 '21 at 15:00
-2

Since I didn’t know the exact date range, only that the last file was today, I ended up using this variation of the top answer, which starts at today’s date and iterates backwards one day at a time until a download fails (presumably with a 404):

d=$(date -I);
while wget "http://www.example.com/$d.log"; do
    d=$(date -I -d "$d - 1 day");
done
Hjulle
  • 2,471
  • 1
  • 22
  • 34
  • Just fyi, people don't like when you answer your own question with a quick n dirty solution when some research might give you a more refined approach – pythonian29033 Feb 15 '23 at 02:51
  • @pythonian29033 that’s the solution I used *after* reading the top answer though, since this was a slight AB problem. i didn’t know the exact date range, so fetching until 404 was the best for me – Hjulle Feb 16 '23 at 12:06