Run multiple Wget commands simultaneously

Question

I have a text file that has a url and the desired file name on each line (seperated by a space). I am looping through this text file and downloading the url and saving it as the desired name using wget:

while IFS=' ' read a b
 do wget $a -O $b
done < list.txt

The problem is my list contains almost 9000 files so downloading them one by one will take a long time. Is there anyway I can download them simultaneously?

a google search on `bash limit number of background jobs` brings up several hits like [this](https://stackoverflow.com/q/1537956), [this](https://stackoverflow.com/q/38774355), [this](https://stackoverflow.com/q/38160), [this](https://superuser.com/q/153630), [this](https://stackoverflow.com/q/64694448) and [this](https://stackoverflow.com/q/49740746); several approaches addressed in the various answers ... some of them should work for this scenario — markp-fuso, Sep 17 '21 at 16:29
@LéaGris I am not sure this code seperates the url from the filename as I get an error saying that the file was not found because it had just combined each line — M9A, Sep 19 '21 at 16:11

score 1 · Accepted Answer · answered Sep 17 '21 at 16:30

1

You can do something like :

number=10
while IFS=' ' read a b; do 
    if  test $(jobs | grep Running | wc -l) -ge $number; then
        sleep 1
    else
        echo "Starting $a..."
        wget $a -O $b &
    fi
done < list.txt
wait

answered Sep 17 '21 at 16:30

Philippe

20,025
2
23
32

score 0 · Answer 2 · answered Sep 17 '21 at 15:55

0

Try with:

while IFS=' ' read a b; do 
  wget $a -O $b &
done < list.txt

# Wait for background jobs to terminate...
wait

PS: Keep in mind that 9000 concurrent connections maybe not handled by the server, so I suggest you to limit the connection number in chunks.

answered Sep 17 '21 at 15:55

Antonio Petricca

8,891
5
36
74

I am not looking to do 9000 concurrent connections but maybe a small number like 10. How can I limit it to 10? – M9A Sep 17 '21 at 16:00
You can limit them by placing a counter into your loop. You have to code it on your own. – Antonio Petricca Sep 17 '21 at 16:01

Run multiple Wget commands simultaneously

2 Answers2