-1

There is this S3 notification feature described here:

Amazon S3 event notifications are designed to be delivered at least once. Typically, event notifications are delivered in seconds but can sometimes take a minute or longer.

and discussed here.

I thought I could mitigate the duplications a bit by deleting files I have already processed. The problem is, when a second event to the same file comes (a minute later) and I try to access the file, I don't get an HTTP 404, I get an ugly AccessDenied:

[ERROR] ClientError: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 111, in lambda_handler
    raise e
  File "/var/task/lambda_function.py", line 104, in lambda_handler
    response = s3.get_object(Bucket=bucket, Key=key)
  File "/var/runtime/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)

which is unexpected and not acceptable.

I don't want my lambda to suppress AccessDenied errors for obvious reasons. Is there an easy way to find out if the file has been already processed in the past or if notification service is playing tricks?

EDIT:

For those who think this is "an indication of some bug in my application" here the relevant piece of code:

    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')

    logger.info(f'Requesting file from bucket {bucket} with key {key}')

    try:
        response = s3.get_object(Bucket=bucket, Key=key)
    except ClientError as e:
        error_code = e.response["Error"]["Code"]
        if error_code == 'NoSuchKey':
            logger.info('Object does not exist any more')
            return
        else:
            raise e

It rather smells like an ugly issue on AWS side to me.

Marek Puchalski
  • 3,286
  • 2
  • 26
  • 35
  • 4
    First thing I would say is that duplicate event notifications from a single S3 event are rare. In fact, if this is happening to you regularly, then it's probably an indication of some bug in your application. It's also unclear to me why your app would work one minute but fail the next minute with AccessDenied for the very same S3 object notification. Your IAM policy presumably did not change during that period of time. Again, this suggests that your application may be bugged or you are misreading the diagnostics here. Is that possible? – jarmod Sep 02 '22 at 14:15
  • I don't see how this can be the case. You have the code above. You have the stack trace too. You need to trust me (and AWS) when we say, that notifications gets duplicated. You have to trust me when I say, that requests to deleted files end with AccessDenied for some reason. This thing happens like once every two weeks. And the reason I see this a lot is probably related to dealing with files this way A LOT. – Marek Puchalski Sep 07 '22 at 09:02
  • 1
    Modify your Lambda function's IAM role policy to include `s3:ListBucket` permission for the S3 bucket in question and then re-test. Suspect that you are seeing 403 AccessDenied when the object does not exist because you don't have `s3:ListBucket` permission. Ordinarily you would see 404. This doesn't help with the potential duplicate notifications from S3 but let's come back to that after fixing the AccessDenied issue. – jarmod Sep 07 '22 at 13:43
  • On the topic of duplicates, see [related question](https://stackoverflow.com/questions/56772299/s3-notification-creates-multiple-events). Do you have any metrics on these duplicates? Like how many you see per 100 events? And a related study [herer](http://www.hydrogen18.com/blog/aws-s3-event-notifications-probably-once.html). – jarmod Sep 07 '22 at 13:49
  • @jarmod: I don't care about the duplicates. I see it more as a feature than a bug. I just want to get rid of AccessDenied problems. If adding s3:ListBucket allows me to fix that, this would be the solution. Unfortunately I will not have time to fix it soon, as some other team is responsible for this and I have no access to their account. And this happens like once every two weeks having tens of thousands of calls succeed in the mean time. Nevertheless, can you write this as an answer? This would be the best pointer so far I believe. – Marek Puchalski Sep 08 '22 at 09:28

2 Answers2

1

On the duplicate delivery of notifications, yes this can happen as documented but is relatively rare:

Amazon S3 event notifications are designed to be delivered at least once. Typically, event notifications are delivered in seconds but can sometimes take a minute or longer.

One possible mechanism to deal with this is to build an idempotent workflow, for example that utilizes DynamoDB to record actions against an object at a given time that can be queried to prevent duplicate workflow on the same object. There are a number of idempotency features in the AWS Lambda PowerTools suite or third-party options that you might consider.

More discussion on the duplicate event topic can be found here.

On the AccessDenied error when attempting to download an absent object that you have GetObject permission for, this is actually a security feature designed to prevent the leakage of information. If you have ListBucket permission then you will get a 404 Not Found response indicating the absence of the object; if you don't have ListBucket then you will get a 403 Forbidden response. To correct this, add s3:ListBucket on arn:aws:s3:::mybucket to your IAM policy.

More discussion on the AccessDenied topic can be found here.

jarmod
  • 71,565
  • 16
  • 115
  • 122
0

You will need to inspect the error code by loading the object using the s3 resource Object to see whether it's a 404. This way you can distinguish between a 404 and a 403 for instance and conclude whether the file has already been deleted in the meantime.

import boto3
import botocore

s3 = boto3.resource('s3')
try:
  s3.Object('my-bucket', 'myfile.txt').load()
except botocore.exceptions.ClientError as e:
  if e.response['Error']['Code'] == "404":
    print('The object does not exist.')
  else:
    print('Something else is wrong.')
    raise

EDIT: Apologies I misread the question. In that case I would just implement idempotency in the processor to make sure you only process each file once.

See for example:

lennart
  • 121
  • 5