4

NOTE: This is not a duplicate, rate limiting uploads and downloads are totally different and many libs I've seen can rate limit downloads but not uploads. This should be reopened.

I'm running a large file upload (2.8 GB) in Python, the code I'm using looks roughly like this:

files = {'md5': ('', md5hash),
         'modified': ('', now),
         'created': ('', now),
         'file': (os.path.basename(url), fileobject, 'application/octet-stream')}
m = requests_toolbelt.MultipartEncoder(fields=files)
headers['content-type'] = m.content_type
r = s.post(url, data=m, params=params, headers=headers)

When I run this code, everything else on my network just stops working. Websites stop working, etc. My guess would be that Python is saturating my router's TCP packet buffer or something. This behavior will be well known to anyone who has tried running a BitTorrent client without specifying limits on their upload speed.

Is there any way I can throttle the upload speed so my users won't have their network destroyed?

seth
  • 2,741
  • 3
  • 20
  • 15
Alex
  • 3,429
  • 4
  • 36
  • 66
  • These answers seem to have some information about libraries to use: http://stackoverflow.com/questions/5888139 http://stackoverflow.com/questions/3488616 – Moberg Jan 29 '15 at 14:21
  • Probably it is saturating the upload channel, which tends to be narrower on most consumer broadband providers. – Paulo Scardine Jan 29 '15 at 14:22
  • 2
    NOT a duplicate...rate limiting download and upload speed are not the same thing. In particular, without library support I can't rate limit a download because I have no control over the data being sent. HOWEVER on an upload I can be sneaky and pass in a lazy version of the data to be uploaded and limit the rate I allow that data to be read by requests – Peter Gerdes Mar 06 '17 at 07:12
  • This is not a duplicate, rate limiting uploads and downloads are totally different and many libs I've seen can rate limit downloads but not uploads. This should be reopened. – seth Oct 23 '20 at 18:28
  • Not a duplicate, my answer is this can be done with [requests toolbar](https://toolbelt.readthedocs.io). [Code is here since I can't format in a comment.](https://gitlab.com/-/snippets/2212960) The upload will stream. The callback inserts a delay. When tested in locust, it seemed to slow down requests. Burst of upload/pause may not be desired. – Bufke Nov 28 '21 at 20:44
  • SO has a thing about closing questions as duplicates when they're not... – Alex Dec 01 '21 at 05:49

0 Answers0