How to monitor progress of a HTTP PUT upload using Python Requests and Clint

Question

I am writing a simple commandline application - transfer.py - to allow for uploading and downloading files from the transfer.sh service as a learning exercise, using the 'requests' library for HTTP. Thanks to some answers on here, I was able to implement a progress bar using python-clint and python-requests for monitoring the file download - said functionality being seen here.

Anyway, I got very, very lost when trying to implement the same kind of progress bar to monitor the upload - which uses HTTP PUT. I understand conceptually it should be very similar, but cannot for some reason figure it out, and would be very thankful if someone could point me in the right direction on this. I tried a few methods using multipart encoders and suchlike, but those lead to the file being mangled on the way up (the service accepts raw PUT requests, and multipart encoding messes it up seemingly).

The end goal is to write a script to AES encrypt the file to be uploaded with a random key, upload it to the service, and print a link + encryption key that can be used by a friend to download/decrypt the file, mostly for fun and to fill in some knowledge-gaps in my python.

score 2 · Answer 1 · edited Jul 26 '16 at 01:58

I recommend you use the requests_toolbelt with the clint.textui.progress module. I found this code which will do.

from clint.textui.progress import Bar as ProgressBar
from requests_toolbelt import MultipartEncoder, MultipartEncoderMonitor

import requests


def create_callback(encoder):
    encoder_len = encoder.len
    bar = ProgressBar(expected_size=encoder_len, filled_char='=')

    def callback(monitor):
        bar.show(monitor.bytes_read)

    return callback


def create_upload():
    return MultipartEncoder({
        'form_field': 'value',
        'another_form_field': 'another value',
        'first_file': ('progress_bar.py', open(__file__, 'rb'), 'text/plain'),
        'second_file': ('progress_bar.py', open(__file__, 'rb'),
                        'text/plain'),
        })


if __name__ == '__main__':
    encoder = create_upload()
    callback = create_callback(encoder)
    monitor = MultipartEncoderMonitor(encoder, callback)
    r = requests.post('https://httpbin.org/post', data=monitor,
                      headers={'Content-Type': monitor.content_type})
    print('\nUpload finished! (Returned status {0} {1})'.format(
        r.status_code, r.reason
        ))

For what it's worth, the above code does a POST (not a PUT). My very brief investigation so far suggests it doesn't work (at the very least, the progress callback doesnt get called) with a PUT. — Tom Dalton, Aug 08 '19 at 09:27
you are right. I dont know how 3 people could upvote something that does'nt answer the question. — grill05, Jan 24 '20 at 16:20

score 0 · Answer 2 · answered Aug 03 '22 at 19:50

The following code should work for you:

import requests
import os
from tqdm import tqdm
from tqdm.utils import CallbackIOWrapper

def upload_from_file(src, dst):
    file_size = os.path.getsize(src)
    with open(src, "rb") as fd:
        with tqdm(desc=f"Uploading", total=file_size, unit="B", unit_scale=True, unit_divisor=1024) as t:
            reader_wrapper = CallbackIOWrapper(t.update, fd, "read")
            response = requests.put(dst, data=reader_wrapper)
            response.raise_for_status()

SRC = '/path/to/file'
DST = '/url/to/upload'

upload_from_file(SRC, DST)

Just define your own SRC and DST variables.
Then you can just copy and paste the code.
You can try to use DST='http://httpbin.org/put' for testing.

Enjoy!

How to monitor progress of a HTTP PUT upload using Python Requests and Clint

2 Answers2