Picking file path and file name with async file download

Question

I am currently using this code (python 3.5.2):

from multiprocessing.dummy import Pool
from urllib.request import urlretrieve

urls = ["link"]
result = Pool(4).map(urlretrieve, urls)
print(result[0][0])

It works, but gets saved to the temp file with some weird name, is there a way to pick a file path and possibly a file name? as well as adding a file extension, it gets saved without one.

Thanks!

score 0 · Accepted Answer · edited May 23 '17 at 11:46

0

You simply need to supply a location to urlretrieve. However pool.map doesn't appear to support multiple args in functions (Python multiprocessing pool.map for multiple arguments). So, you can refactor, as described there, or use a different multiprocessing primitive, e.g. Process:

from multiprocessing import Process
from urllib.request import urlretrieve

urls = ["link", "otherlink"]
filenames = ["{}.html".format(i) for i in urls]
args = zip(urls, filenames)
for arg in args:
    p = Process(urlretrieve, arg)
    p.start()

In the comments you say you only need to download 1 url. In that case it is very easy:

from urllib.request import urlretrieve
urlretrieve("https://yahoo.com", "where_to_save.html")

Then the file will be saved in where_to_save.html. You can of course provide a full path there, e.g. /where/exactly/to/save.html.

edited May 23 '17 at 11:46

Community

1
1

answered Dec 24 '16 at 09:53

rofls

4,993
3
27
37

I did get a raise URLError('unknown url type: %s' % type) urllib.error.URLError: – Jelly bean Dec 24 '16 at 10:08
Actually i was getting errors too.I thought that would have worked. See edit. – rofls Dec 24 '16 at 10:08
The core of your issue is that you need to pass a filename to `urlretrieve`. – rofls Dec 24 '16 at 10:09
i only need to download 1 url at a time. currently i get this exception AssertionError: group argument must be None for now – Jelly bean Dec 24 '16 at 10:13
@Jellybean if you only need to download one url, then you don't need to use `Pool`. – rofls Dec 24 '16 at 23:35

Picking file path and file name with async file download

1 Answers1