3

Python's urllib.request module offers a urlopen function which retrieves a URL's content plus some metadata and stores everything in main memory. In environments with limited memory this can quickly result in MemoryErrors.

There is another function called urlretrieve which seems to do what I am looking for. For some reason, though, the official documentation mentions that it might become deprecated in the future.

Is there an "official", built-in and non-legacy way for performing a download directly to the local file system? I am aware that this can easily be achieved with third-party libraries such as requests but I am working under strict computational & memory constraints and therefore would favor a built-in solution.

zepp133
  • 1,542
  • 2
  • 19
  • 23
  • Does this answer your question? [Download large file in python with requests](https://stackoverflow.com/questions/16694907/download-large-file-in-python-with-requests) – zvone Dec 23 '19 at 21:11
  • @zvone Thanks for the reference but I am looking for a way to achieve the same thing without the requests library – zepp133 Dec 23 '19 at 21:28
  • @zepp.lee Why don't you want to use requests? – AMC Dec 23 '19 at 22:12
  • @AMC I am writing code to run on very limited (embedded) hardware and therefore every saved dependency counts – zepp133 Dec 23 '19 at 22:26
  • _run on very limited (embedded) hardware_ I'm confused. – AMC Dec 23 '19 at 22:39

1 Answers1

3

If you want to limit yourself to Python's standard library, please note that urlopen returns HTTPResponse objects, which have methods to read the response into memory chunk by chunk. You can buffer chunks of the response in RAM and write it to disk or elsewhere along the way.

The requests module makes the whole thing more streamlined.

9000
  • 39,899
  • 9
  • 66
  • 104
  • 3
    To boot: `urllib.request.urlretrieve` [does exactly this](https://github.com/python/cpython/blob/0846e5d4603434c2bbf8a528677cf1ff3fe29b95/Lib/urllib/request.py#L229), in blocks of 1024*8 – Brad Solomon Dec 23 '19 at 21:14
  • @BradSolomon Very interesting, thank you for the link! Do you by any chance happen to know why there are plans to deprecate `urlretrieve`? – zepp133 Dec 23 '19 at 21:31