3
  • My average download speed on the wifi I'm currently using runs up to 200 Megabits/s.

I noticed that when downloading an .mp4 file with requests:

My Code:

with open(path, "wb") as f:
    for chunk in res.iter_content(1024):
        f.write(chunk)

In my Task Manager, it shows Python using only 8 Megabits/s on average when downloading this file. I was just wondering if Python or Requests throttles download speeds by default when it comes to larger files.

Koyomi-Oniichan
  • 89
  • 6
  • no other application or program is using the internet at the same time? – Charalamm Dec 12 '20 at 06:00
  • 1
    have you tried fetching it via wget or curl? the server for that particular resource may be doing the throttling. – JL Peyret Dec 12 '20 at 06:04
  • @Charalamm Just discord but it isn't very bandwidth heavy. Ultimately, Python seems to cap it at 8Mb/s no matter what I try. I've even tried changing the `chunk_size` to something massive but that wouldn't make a difference. – Koyomi-Oniichan Dec 12 '20 at 06:05
  • @JLPeyret I haven't tried `curl` yet. How would I do so? Is `curl` a python package or would I have to call from subprocess? Are you referring to `pycurl`? – Koyomi-Oniichan Dec 12 '20 at 06:06
  • 1
    1024 bytes is very small and gives much overhead. A modern computer should be able to process much larger chunks . Try chunks of 10000, 50000 ... and look at the result. – Mace Dec 12 '20 at 06:07
  • 1
    Try taking a multithreaded approach, that's the way to do it. – l'L'l Dec 12 '20 at 06:08
  • @l'L'l how could I do so? I've never worked with `multi-threading` before. – Koyomi-Oniichan Dec 12 '20 at 06:09
  • [This would be the basic idea](https://stackoverflow.com/a/33543751/499581), although there are tons of other ways to do it. – l'L'l Dec 12 '20 at 06:14
  • @l'L'l How does this work? Is this just splitting the content of the download and using asynchronous requests? – Koyomi-Oniichan Dec 12 '20 at 06:17
  • curl is just a command line utility on Linux or macos to grab a file. For example, after copying the address for the Subway image at https://www.bbc.com/news/technology-55279051, I just used `curl https://ichef.bbci.co.uk/news/976/cpsprodpb/52A1/production/_116035112_gettyimages-1175368406.jpg --output foo.jpg` Now, if you point it to your file's URL and curl is also showing 8Mb/s then it's a server-side throttle, not a `requests` issue. Now, if you're on Windows I don't know what to use, though Powershell seems to have some equivalent. – JL Peyret Dec 12 '20 at 06:25

1 Answers1

1

Ok, it seems that the host that I was downloading from was limiting the bandwidth for their public downloads to 1MB. I've done some research on if Python limits bandwidth and I'm positive that it does not.

Koyomi-Oniichan
  • 89
  • 6