1

Background

I have a program where a client can see a list of files on the server, and then choose a file to upload from the server to Google Cloud Storage. While a file is uploading from the server to Google Cloud Storage, I want to let the client know what the progress of the file upload is. So far I'm able to show the progress of the upload on the server's terminal screen using the wrapattr method of tqdm, as shown below:

object_address = str(uuid.uuid4())
upload_url, upload_method = get_upload_url(object_address) # fetches signed upload URL 
size = os.path.getsize(filename)
with open(filename, "rb") as in_file:
    total_bytes = os.fstat(in_file.fileno()).st_size
    with tqdm.wrapattr(
        in_file,
        "read",
        total=total_bytes,
        miniters=1,
        desc="Uploading to my bucket",
    ) as file_obj:
        response = requests.request(
            method=upload_method,
            url=upload_url,
            data=file_obj,
            headers={"Content-Type": "application/octet-stream"},
        )
        response.raise_for_status()

return object_address, size

Problem

I need to send the progress of the upload to the client. It's not helpful if you can only see the progress of the upload on the server's terminal screen.

Question

How can I retrieve the progress of the upload so that I can send it to the client?

Unfortunately, I don't think the wrapattr function has any parameters for callback functions that I could call every time a chunk is uploaded to Google Cloud Storage. I'm open to other ways I could retrieve progress without using the tqdm wrapattr method.

I know Google Cloud Storage has resumable uploads that allow you to track progress, but they use cURL so I would have to use a subprocess and I would rather use something cleaner than that.

Thanks!

jjasper
  • 191
  • 1
  • 9
  • I can see you are only sharing the code from your app, but how would the client interact with the app itself? Can you please add more information about how the users would get the information you are trying to show? I think you are planning to use the terminal as well, but I think you should share a little more information on this. – Alex Oct 28 '21 at 17:18
  • 1
    @Ginger - Thanks for the feedback. I updated the background to provide some more detail. A client will see a list of files on the server. Then the client will choose a file on the server to upload from the server to Google Cloud Storage, and the server will do so. Again, the problem is getting the upload progress back to the client while the file is uploading. Thanks! – jjasper Nov 10 '21 at 15:17

1 Answers1

0

As I can see, you are already able to get the progress and display it successfully, but not on the client side. In this question they use the requests library to get the progress, trying to emulate a file adaptor similar to the one from urllib2 POST progress monitoring:

class IterableToFileAdapter(object):
    def __init__(self, iterable):
        self.iterator = iter(iterable)
        self.length = len(iterable)

    def read(self, size=-1): # TBD: add buffer for `len(data) > size` case
        return next(self.iterator, b'')

    def __len__(self):
        return self.length

You could adapt this method to your own app using tqdm.

Alex
  • 778
  • 1
  • 15