A bash script for multiple wget commands

Question

I have mutliple (over 100) similar to this:

wget -q -nH --cut-dirs=5 -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2009_02//00031327004/auxil/
wget -q -nH --cut-dirs=5 -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2009_01//00031327001/uvot/
wget -q -nH --cut-dirs=5 -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2010_12//00031856009/uvot/
wget -q -nH --cut-dirs=5 -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2008_01//00031043003/uvot/
wget -q -nH --cut-dirs=5 -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2012_01//00032237004/uvot/

I have been told that this can be completed very quickly and easy with a bash script, could someone give me an example if possible for multiple wget commands? What do I need to include within the script? Apologies for my n00b questions, but...I am one!

How similar? What's different between each call? Just the URL? — Mr. Llama, Dec 03 '15 at 20:50
`bash your-file-here.txt`, if all you want is to run it as a script. — Charles Duffy, Dec 03 '15 at 21:31

miken32 · Answer 1 · 2015-12-03T21:21:53.210

Assuming your URL list looks like this:

http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2009_02//00031327004/auxil/
http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2009_01//00031327001/uvot/
http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2010_12//00031856009/uvot/
http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2008_01//00031043003/uvot/
http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2012_01//00032237004/uvot/

Just read your URLs from the text file using a while loop:

#!/bin/bash
while read url; do
    wget -q -nH --cut-dirs=5 -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks "$url"
done < urls.txt

score 0 · Accepted Answer · answered Dec 03 '15 at 20:52

0

Assuming your noob question is about speeding up the process time,

you can spawn multiple process's in background & and wait for them

wget firsturl &
wget secondurl &
...
wait

answered Dec 03 '15 at 20:52

Rozuur

4,115
25
31

I'd really prefer a bash script. – GCien Dec 03 '15 at 20:53
its a bash script, just replace firsturl with what ever you are using – Rozuur Dec 03 '15 at 20:54
And this can be placed in a text file to be run as a script? – GCien Dec 03 '15 at 20:56
yes, just replace ... with all your urls and end each line with `&` – Rozuur Dec 03 '15 at 20:58

A bash script for multiple wget commands

2 Answers2