Callback with Boto3, Showing Upload Progress

Question

I am in a situation with a document storage feature in a Django/Python application (using Boto3) where I also need to provide the user with information on the progress of an upload.

The methods I have seen which use straight JS are not all universally 'usable' across browsers, what I want is to be able upload a file using Boto3 and then also show an upload progress bar in my Django template.

Looking at the docs here: https://boto3.amazonaws.com/v1/documentation/api/latest/_modules/boto3/s3/transfer.html I am not understanding how it would be possible to show building progress of the upload calling ProgressPercentage() from within a template.

How would I go about displaying the actual progress for my user in a template using Boto3?

class ProgressPercentage(object):
        def __init__(self, filename):
            self._filename = filename
            self._size = float(os.path.getsize(filename))
            self._seen_so_far = 0
            self._lock = threading.Lock()

        def __call__(self, bytes_amount):
            # To simplify we'll assume this is hooked up
            # to a single filename.
            with self._lock:
                self._seen_so_far += bytes_amount
                percentage = (self._seen_so_far / self._size) * 100
                sys.stdout.write(
                    "\r%s  %s / %s  (%.2f%%)" % (
                        self._filename, self._seen_so_far, self._size,
                        percentage))
                sys.stdout.flush()


transfer = S3Transfer(boto3.client('s3', 'us-west-2'))
# Upload /tmp/myfile to s3://bucket/key and print upload progress.
transfer.upload_file('/tmp/myfile', 'bucket', 'key',
                         callback=ProgressPercentage('/tmp/myfile'))

score 2 · Answer 1 · answered Sep 09 '20 at 09:47

There are three related answers in SO:

Get progress callback in aws boto3 uploads (couldn't make it work, but seems elegant)
Track download progress of S3 file using boto3 and callbacks (this one is for downloads, but helped me understand what the callback should look like)
S3 Python Download with Progress Bar (worked, I used this one to write my code).

Here is what I did that worked:

import boto3
from botocore.exceptions import NoCredentialsError, ClientError
import os
import progressbar

ACCESS_KEY = 'MY_ACCESS_KEY'
SECRET_KEY = 'MY_SECRET_KEY_THAT_I_KNOW_SHOULD_NOT_BE_HERE_BUT_I_HAVENT_IMPLEMENTED_THE_PROPER_WAY_YET_ALSO_I_DO_NOT_KNOW_WHY_I_AM_YELLING_SORRY'

def upload_to_aws(local_file, s3_bucket, s3_folder, s3_filename,force_overwrite='n'):
    s3 = boto3.client('s3', aws_access_key_id=ACCESS_KEY,
                    aws_secret_access_key=SECRET_KEY)


    def write_to_aws():

        statinfo = os.stat(local_file)
        up_progress = progressbar.progressbar.ProgressBar(maxval=statinfo.st_size)
        up_progress.start()

        def upload_progress(chunk):
            up_progress.update(up_progress.currval + chunk)


        try:
            print("Writting")
            s3.upload_file(local_file, s3_bucket, s3_folder+s3_filename, Callback=upload_progress)
            print("Upload Successful")
            return True
        except FileNotFoundError:
            print("The source file was not found")
            return False
        except NoCredentialsError:
            print("Credentials not available")
            return False
    try:
        s3.head_object(Bucket=s3_bucket, Key=s3_folder+s3_filename)
        if force_overwrite=='y':
            write_to_aws
        else:
            ask_overwrite = input('File already exists at destination. Overwrite?')
            if ask_overwrite == 'y':
                write_to_aws()
            else:
                print('leaving application')
    except ClientError as e:
        write_to_aws()

I know you probably figured that out already, but if anyone else finds this while searching for the solution, I hope this helps

Callback with Boto3, Showing Upload Progress

1 Answers1