Downloading list of urls/files using loop - python

Question

I need to download approximately 1000 file/url and it will be hard to download them manually.

I tried to put the urls in a list and loop through the list but it I think my code overwrite the previous files and keep only the last item in the list

Here is my code

#!/usr/bin/env python

import urllib3
http = urllib3.PoolManager()

urls = ["http://url1.nt.gz" , "http://url2.nt.gz" , "http://url3.nt.gz"]
N =1; // counter helps me to rename the downloaded files
print "downloading with urllib"
for url in urls
 r = http.request('GET',url)
 Name =str(N+1) // each time increment the counter by one 
 with open("file"+Name+".nt.gz", "wb") as fcont:
                fcont.write(r.data)

Any suggestions?

Patrick Artner · Answer 1 · 2017-11-25T18:58:20.347

1

You do not increment the counter - you add 1 without saving it back to N

Add N += 1 after setting Name. You are missing a : after your for.

I am not quite sure where you have your 1000s of urls - I only see 3 in urls.

#!/usr/bin/env python

import urllib3
http = urllib3.PoolManager()

urls = ["http://url1.nt.gz" , "http://url2.nt.gz" , "http://url3.nt.gz"]
N =1; // counter helps me to rename the downloaded files
print "downloading with urllib"
for url in urls:
    r = http.request('GET',url)
    Name =str(N+1) 
    N += 1
    with open("file"+Name+".nt.gz", "wb") as fcont:
        fcont.write(r.data)

edited Nov 25 '17 at 18:58

answered Nov 25 '17 at 18:53

Patrick Artner

50,409
9
43
69

Thank you! just for simplicity I added only 3 urls on my list. Is there any simple way to read the list elements from a txt file? Oh my God, I can't imagine that I spent two days figuring out what is the problem on my code and its super simple! – saad Nov 25 '17 at 21:12
Look at [read line from gzip file](https://stackoverflow.com/a/30868178/7505395) for reading from a *.gz line by line. Mark as answer if this question was answered :) – Patrick Artner Nov 25 '17 at 22:00
@saad Sorry, forgot to add your name to last comment – Patrick Artner Nov 25 '17 at 22:06

score 0 · Accepted Answer · answered Nov 25 '17 at 18:59

0

print "downloading with urllib" for url in urls r = http.request('GET',url) Name += N

answered Nov 25 '17 at 18:59

Harry_Hirsch

28
3

Downloading list of urls/files using loop - python

2 Answers2