1

I would like to download multiple files from the linux command line, the link structure is stimilar to this https://archive.org/compress/tmg2002-10-23/formats=Flac,Metadata,Text,Checksums,Flac%20FingerPrint and the variable part is in the middle /tmg2002-10-23/with different dates. How can I use a download manager such as wget/aria2 to download all of the files? I tried with /tmg*/ in both wget/aria2 but it just doesn't work

user3070484
  • 15
  • 3
  • Well, using `*` in Bash makes the shell think that this is a file in the filesystem so it will try to do some kind of filename expansion. This cannot work. What you need is an HTML page residing on that server from which you might be able to scrape the available dates. Alternatively, if you are more or less sure about the date range you may just want to try to enumerate the days in that range, simply try to load them, and ignore the days with no hit. But, that's pretty much brute force. – Marcus Rickert Dec 19 '13 at 23:16

2 Answers2

1

Adapted from here.

for d in {0..365}
do
    dt=$(date -d "2012-01-01 + $d days" +'%Y-%m-%d')
    wget "https://archive.org/compress/tmg$dt/formats=Flac,Metadata,Text,Checksums,Flac%20FingerPrint"
done

Change "2012-01-01" for the date you want to start and change 365 for the number of days to download after the starting date. The date function already deals with leap years for you.

Community
  • 1
  • 1
Fábio Perez
  • 23,850
  • 22
  • 76
  • 100
0

For example

for i in "2002-10-23" "2002-10-22"
do wget "https://archive.org/compress/tmg${i}/formats=Flac,Metadata,Text,Checksums,Flac%20FingerPrint"
done

You should include right options for wget. And of course the script could be much more sophisticated.

Petr Matousu
  • 3,120
  • 1
  • 20
  • 32