2

My gut feel is no. That request.post is blocking by nature. But I may be wrong and even if I'm right I'd love to work out what alternatives there are. Essentially I have a file upload request that I could see in the Firefox Network Console and copy as a fetch, and then reproduce in python using request.post.

It works fine. Uploads the file dandily.

But I'd like a progress bar.

I am using the python3-wget package to download it and it has a progress bar. Neat. A quick look inside and it seems a tad involved, using urllib, not the request package. Here's the code I'm using:

BaseURL =        result["BaseURL"]
Signature =      result["Signature"]
GoogleAccessId = result["AccessKey"]["Value"]
CacheControl =   result["AdditionalBody"][0]["Value"]
Policy =         result["PolicyBase64"]
ACL =            result["ACL"]
KeyPrefix =      result["KeyPrefix"]

Type = os.path.splitext(local_file)[1][1:]
Checksum = api.check_sum(local_file)

if Type:     
    Key = "{}{}/{}.{}".format(KeyPrefix, Type, Checksum, Type)
else:
    Key = "{}{}/{}.{}".format(KeyPrefix, "unknown", Checksum, "unknown")

Size = os.path.getsize(local_file)

parts = [
    ('key', (None, Key)),
    ('acl', (None, ACL)),
    ('policy', (None, Policy)),
    ('signature', (None, Signature)),
    ('GoogleAccessId', (None, GoogleAccessId)),
    ('Cache-control', (None, CacheControl)),
    ('Content-Type', (None, MimeTypeOfFile)),
    ('file', (os.path.basename(local_file), open(local_file, 'rb'), MimeTypeOfFile))
]

heads = {"ngsw-bypass": "1", "x-client-data": "somelongencodedstring"}

response = requests.post(BaseURL, files=parts, headers=heads)
  • result is returned by another request which provides all the upload details needed to this Google Cloud Service (which is where BaseURL points).
  • local_file is just the name of the file I'm uploading.
  • somelongencodedstring is an API key of sorts by look of it, I just copied it from the fetch request logged by Firefox and have anonymised it here.

What I'm hoping to do is replace that last line (and whatever else I need to back it up with) with something that has a callback for rendering a progress bar.

I can see that the wget implementation in Python uses urllib.request.FancyURLopener in and its retrieve method which has an argument that accepts a function as a callback.

Am I doomed to dig into the innards of urllib? Or can request.post do something for me? Or is there some other ready solution to slip in where request.post is?

Bernd Wechner
  • 1,854
  • 1
  • 15
  • 32
  • Does this answer your question? https://stackoverflow.com/questions/41106599/python-3-5-urllib-request-urlopen-progress-bar-available – GProst Sep 30 '20 at 12:49
  • Might do, that example is for downloads, and I might learn something from it. Had a quick look for now ... more later. – Bernd Wechner Sep 30 '20 at 23:56

1 Answers1

0

I would suggest looking at aiohttp client. Particularly, you can specify TraceConfigs for a request and add on_response_chunk_received callbacks. For details see these docs: https://docs.aiohttp.org/en/stable/tracing_reference.html

Some examples here: https://docs.aiohttp.org/en/stable/client_advanced.html#client-tracing

GProst
  • 9,229
  • 3
  • 25
  • 47