How to trigger an AWS Event Rule when a S3 key with a specific suffix gets uploaded

Question

I'm trying to create an AWS Event Rule that is only triggered when a file with a specific suffix is uploaded to an S3 bucket.

{
  "source": [
    "aws.s3"
  ],
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "detail": {
    "eventSource": [
      "s3.amazonaws.com"
    ],
    "eventName": [
      "PutObject",
      "CompleteMultipartUpload"
    ],
    "requestParameters": {
      "bucketName": [
        "bucket-name"
      ],
      "key": [
        { "suffix": ".csv" }
      ]
    }
  }
}

As I understand there AWS has content-based filtering which can be used but docs doesn't show the ability to use a suffix, only prefix among other patterns: https://docs.aws.amazon.com/eventbridge/latest/userguide/content-filtering-with-event-patterns.html

Ideally I could be able to do it here without the need for an intermediary Lambda as my event target is an ECS Fargate task.

Chris Williams · Answer 1 · 2020-07-11T11:02:15.237

6

At this time (July 2020) CloudWatch events does not appear to have suffix filtering built into it.

You could instead configure an S3 Event Notification which do support the ability to specify prefixes and suffixes.

By using an S3 event notification you can still have your target as a Lambda.

edited Jul 11 '20 at 11:02

answered Jul 11 '20 at 10:51

Chris Williams

32,215
4
30
68

Hi Chris, my target is actually an ECS Fargate task and I do **not** want to have an intermediary Lambda. – Aaron Zhong Jul 11 '20 at 10:54
@AaronZhong at the moment there is no option to use suffixes with content filtering (its relatively new) for CloudWatch events. Hopefully in the future we'll see this added. The alternative is to have all the files you'd want to pass in a specific prefix so that you can then have your fargate task as a direct target. – Chris Williams Jul 11 '20 at 11:01
@ChrisWilliams Do we have this feature now to add suffix in cloudwatch rule? – Abdul Haseeb Aug 24 '21 at 08:51
my only concern is that my target is getting triggered event if a new empty folder is created. I just want to avoid this. – Abdul Haseeb Aug 24 '21 at 08:51
2

You could go S3 event -> SNS -> Eventbridge -> Fargate – Merlin Oct 08 '21 at 02:26
2

@Merlin though it will work but my AWS cost will increase quite a bit as I do not want 500 million messages extra SNS to be added every month Unbelievable they do not support proper filtering - do they want people to use less filters so more events get fired and more money they make? – Sun Aug 14 '22 at 14:25
@Sun Maybe there is a difficulty in how filtering is implemented. At any rate, you could use a prefix like `s3://yada-yada/csv/path/file.csv` and land all of your CSV files into a prefix containing 'csv'. Possibly also tagging the object in the upload could be a trigger. – Merlin Aug 15 '22 at 15:45

score 1 · Accepted Answer · answered Feb 25 '23 at 00:14

1

It appears that suffix matching has been added

https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns-content-based-filtering.html#eb-filtering-suffix-matching

Therefore, the example given in the question is now valid!

answered Feb 25 '23 at 00:14

Aaron Zhong

921
8
22

How to trigger an AWS Event Rule when a S3 key with a specific suffix gets uploaded

2 Answers2