0

I'm attempting to make a file that downloads a large list of files from a website. The download function I've got is this:

def download(files, downloadDirectory, rootDirectory):
import urllib.request

for file in files:
    urllib.request.urlretrieve(rootDirectory + file, downloadDirectory + file)

How can I go about adapting this to download the files in parallel so that the speed is maximized? Is it best to use multithreading, or is there something else I can use? Other questions that answer this don't seem to be for python 3.4.3.

xyz
  • 113
  • 9
  • Possible duplicate: http://stackoverflow.com/questions/16219389/download-many-files-in-parallel-linux-python – tomasyany Aug 01 '15 at 03:38
  • And this: http://stackoverflow.com/questions/30208345/parallel-downloads-using-pysmartdl-python-multiprocess – tomasyany Aug 01 '15 at 03:38
  • pySmartDL can do multiple simultaneous non-blocking downloads and max out available bandwidth but I did not get consistent results. A few times it crashed with urllib socket timeouts. Another time it failed after two HTTP read retry attempts after reading the entire file each time while firefox downloaded it with no problems. –  Aug 01 '15 at 06:17

0 Answers0