4

For example I want to download data from: http://nimbus.cos.uidaho.edu/DATA/OBS/

with the link:

http://nimbus.cos.uidaho.edu/DATA/OBS/pr_1979.nc

to

http://nimbus.cos.uidaho.edu/DATA/OBS/pr_2015.nc

How can I write a script to download all of them? with wget?and how to loop the links from 1979 to 2015?

davejal
  • 6,009
  • 10
  • 39
  • 82
breezeintopl
  • 221
  • 2
  • 14
  • Is the range only 1979-2015, or are you looking for something more dynamic that checks whether a year exists first? – Jeff Puckett May 15 '16 at 00:56
  • Well, yes for this example, I can check that by my eyes for simplicity. So just look for the looping code. Maybe some script like `wget http://nimbus.cos.uidaho.edu/DATA/OBS/pr_*.nc` or `wget .../pr_[1979-2015].nc` – breezeintopl May 15 '16 at 01:02

2 Answers2

5

wget can take file as input which contains URLs per line.

wget -ci url_file

-i : input file
-c : resume functionality

So all you need to do is put the URLs in a file and use that file with wget.

A simple loop like Jeff Puckett II's answer will be sufficient for your particular case, but if you happen to deal with more complex situations (random urls), this method may come in handy.

Community
  • 1
  • 1
Jahid
  • 21,542
  • 10
  • 90
  • 108
  • Thanks you so much for your answer! This actually brings another (two) problem(s) for me for a long time(which I usually do it by hand T. T): 1. How can I create a file with repeated content "...pr_1979.nc \n ...\n ...pr_2015.nc"? 2. How can I create 30+ files with names like "1979.txt" and content "http.../pr_1979.nc"? Well, I think I can use, say R, to concatenate long content strings with variable, say x=1979. But I don't think this is the usual way to do this. Since I am not a computer science major, I am not familiar with how to create a batch of files. (Maybe, geeks use vim to do this?) – breezeintopl May 15 '16 at 14:56
  • OK, so maybe, based on the above answer, it is something like `echo $text1$i$text2 >> filename$i ;` in the loop. – breezeintopl May 15 '16 at 15:11
  • @breezeintopl : yes, but it defeats the purpose (creating each file for each link).`filename$i` should be `filename` so that all links go into the same file. Though I generally extract the URLs from the html source with text editor, generally with [tag:sed]. – Jahid May 15 '16 at 16:31
  • OK, got it. Thank you so much! – breezeintopl May 15 '16 at 23:41
3

Probably something like a for loop iterating over a predefined series.

Untested code:

for i in {1979..2015}; do
  wget http://nimbus.cos.uidaho.edu/DATA/OBS/pr_$i.nc
done
Jeff Puckett
  • 37,464
  • 17
  • 118
  • 167
  • You might also be interested in this answer http://stackoverflow.com/questions/169511/how-do-i-iterate-over-a-range-of-numbers-defined-by-variables-in-bash – Jeff Puckett May 15 '16 at 01:16
  • Thanks! This is very helpful! Do you have any suggested documents or books or tutorials on learning this kind of bash code? – breezeintopl May 15 '16 at 01:31
  • @breezeintopl honestly, not really. you might try posting that as a question on http://softwarerecs.stackexchange.com Personally, I just search stack overflow for anything I need to do, and the answers are most usually here already. – Jeff Puckett May 15 '16 at 01:49