Is boto3.Bucket.upload_file blocking or non-blocking?

Question

i.e. if I were to run the following

bucket = session.Bucket(bucket_name)
bucket.upload_file(Key=s3_key, Filename=source_path)
os.remove(source_path)

Do I have a race condition, depending on the size of the file? Or is upload guaranteed to complete before file deletion?

I recently took a more detailed look at this: [Make Boto3 upload calls blocking (single threaded)](https://stackoverflow.com/a/55850889/7954504) — Brad Solomon, Apr 25 '19 at 13:51

score 3 · Accepted Answer · answered Jun 03 '16 at 16:37

3

The current boto3 upload_file is blocking. As mootmoot said, you should definitely implement some error handling to be safe if you delete the file.

answered Jun 03 '16 at 16:37

Jordon Phillips

14,963
4
35
42

7

Why is this completely opposite of the answer below? Can we get some conclusive evidence? – retrocookie Jun 09 '16 at 23:48

score 2 · Answer 2 · answered Jun 03 '16 at 10:07

Whether blocking or unblocking, you SHOULD NOT rely on the API alone when things went bad. You MUST add exception handling if the upload fail in the middle for any reason(e.g. admin decide to restart the router when you doing the upload).

bucket = session.Bucket(bucket_name)
try :
  bucket.upload_file(Key=s3_key, Filename=source_path)
  os.remove(source_path)
except : 
  raise

Another good practice to upload file to S3 is adding additional Metadata.

bucket.upload_file(
     Key=s3_key, 
     Filename=source_path, 
     extra_args={'Metadata': {'source_path': source_path}}
)

Adding event to S3 Bucket to act on success PUT action also let you create cleanup process if there is success upload but failure on local file removal.(imagine the file is locked or the file is given Read-only access).

score 2 · Answer 3 · answered Jul 21 '17 at 20:19

2

Boto3 does not have support for async calls, so the function is blocking.

See conversations regarding async + boto3 here:

https://github.com/boto/boto3/issues/648

https://github.com/boto/boto3/issues/746

answered Jul 21 '17 at 20:19

Christopher Peter

488
3
5

My evidence is that it is non blocking as I follow an upload loop through files with a delete of the local source files. One in 10 times, the upload fails as source files not found. Yes, I can defensively code against it. But would be good to clear this up definitively to understand what is happenning. – John Curry Nov 03 '20 at 13:04

score 1 · Answer 4 · answered Aug 08 '20 at 10:01

I've made an asynchronous object to upload to S3 and download on your computer. To be sure that in your case for example, file will be deleted after upload, you can use the callback: on_success.

Check it out: https://gist.github.com/fherbine/0d4aa473e5cc2c5f6f8a0e1b35a62625

There's still enhancements to make, but it's working.

Is boto3.Bucket.upload_file blocking or non-blocking?

4 Answers4

Linked