Silent Failure of S3 PutObject?

Question

I have been trying to programmatically upload SNS messages to an S3 bucket using the S3.Object.put() method like so:

bucket_resource=boto3.resource('s3')
bucket_client = boto3.client('s3')
body = subject + message

object = bucket_resource.Object(bucket_name, folder+'/'+fn)
object.put(Body=body)

This has not worked, so I have tried the following to try and upload an object to a particular S3 bucket.

body = subject + message
folder = datetime.datetime.today().strftime('%Y-%m-%d')
fn = datetime.datetime.today().strftime('%H:%M:%S')
key = folder_name + '/' + fn
bucket_resource = s3.Bucket(bucket_name)
bucket.upload_file(body, key)

However, both of these methods are failing silently. I am not getting any access denials, error messages, etc. but I am also not uploading my message to a bucket. I'm not sure what's happening with each invocation of the function, and would appreciate any guidance for people who have successfully uploaded files to buckets programmatically.

Note: I have bucket policies in place where my account is the only account that can put objects in the bucket. Do I need an addendum to give Lambda permission to put objects in the bucket?

It seems unthinkable that try/except, as suggested below, has to be needed in order to prevent exceptions from being silently swallowed. Unhandled exceptions should cause a crash. (right? I mean... right?) How are you comfirming that the files aren't there? Does the value of `key` come back sensible if you log it? Here's a hack to try: Set the bucket name to "uploads". That happens to be a bucket name that AWS has on some kind of blocked or reserved list that always throws exceptions when you try to write to or read from it. If *that* doesn't throw an exception, then I'l be surprised. — Michael - sqlbot, Feb 25 '19 at 22:39
In the second example, shouldn't `bucket.upload_file(body, key)` be `bucket_resource.upload_file(body, key)`? — John Rotenstein, Feb 25 '19 at 22:46
@JohnRotenstein Actually, it was supposed to be bucket_client.upload_file, and that was my oversight. I have a try/except to catch whether a file is successfully uploaded and my logs tell me that despite no key/value errors, I am not getting the file to upload to the S3 bucket. — Bri, Feb 26 '19 at 15:35
In that case, I would recommend starting with a hard-coded `upload_file()` command to confirm that it can work okay (eg `bucket_resource.upload_file('abc','test.txt')`. If that works, then the problem lies with the body or key. If it doesn't work, then you'll need to experiment to figure out where the problem lies. — John Rotenstein, Feb 26 '19 at 20:13
I'm seeing something similar where I'm using `bucket_resource.upload_file()` ... can't figure out what the issue is. — monkut, Jul 09 '20 at 10:22

score 3 · Answer 1 · answered Nov 13 '20 at 14:40

There was no error for me either but I'm using the .NET Core SDK. Turns out my problem was the function finishing before it was able to put the file in S3. In the .NET SDK the PutObject call is Asynchronous (PutObjectAsync) and even though I was waiting for it to finish, turns out I wasn't doing that correctly due to the layers of functions I had.

For anyone struggling with no errors and no files in S3, check that your function isn't finishing before the file is actually uploaded to S3.

score 0 · Answer 2 · answered Dec 29 '21 at 18:50

I have not used python, but I faced a similar issue where my react app crashes when I do s3.upload(). The following answer helped me solve it. https://stackoverflow.com/a/57139331/5295723

I had to convert the base64 image buffer to binary and it worked.

score -1 · Answer 3 · answered Feb 25 '19 at 20:59

-1

See here on handling errors in python.

Basically try/catch exception around sdk call, something along the lines of:

import boto3
from botocore.exceptions import ClientError, ParamValidationError

try:
    bucket_resource=boto3.resource('s3')
    bucket_client = boto3.client('s3')
    body = subject + message
    object = bucket_resource.Object(bucket_name, folder+'/'+fn)
    object.put(Body=body)
except s3.exceptions.<fill in>:
    print("known error occured")
except ClientError as e:
    print("Unexpected error: %s" % e)

This is the link the original poster provided, and it shows possible exceptions you can catch for an S3 client

answered Feb 25 '19 at 20:59

Deiv

3,000
2
18
30

2

That isn't applicable to my post. I am not getting any errors whatsoever, and I believe I posted that in my original question. I'm not getting errors, but an object is also not being created. – Bri Feb 25 '19 at 21:38
Yes you aren't getting any errors because you aren't catching any, that's why they're silent – Deiv Feb 25 '19 at 21:39
I feel like Cloudwatch would tell me if my lambda failed due to an error I wasn't catching, though. And Cloudwatch reports that my function successfully executes. If there is any error in a Lambda function, the function fails. – Bri Feb 25 '19 at 21:40
Not necessarily. Unless you are logging the error, it wouldn't log it because it's assuming you don't want to see it. Same goes for whether or not you want the function to fail if a piece of code fails in it. Lambda doesn't fail if you handle an error and tell it not to, or if you don't handle an error and tell it nothing it will assume it's job is done. – Deiv Feb 25 '19 at 21:44
My personal experience is mainly with nodejs lambdas, but I'm assuming it is the same behavior for all based on that experience. Try using a try/catch and see what happens, it's worth a shot – Deiv Feb 25 '19 at 21:45

Silent Failure of S3 PutObject?

3 Answers3