Get AWS request API key in S3 bucket trigger?

Question

As part of an asynchronous content handling system, we've defined an API Gateway method bound to the S3 integration service that forwards POST requests to an S3 bucket. We then use an S3 event trigger to an ObjectCreated event to an SQS queue, which is eventually consumed by a Lambda function for subsequent processing.

We have a new processing requirement that requires us to identify the API Key ID used to authenticate the original APIGW request in the SQS handler lambda, but we've not come up with an approach. We looked for a way to pass the Authorization header of the client request as part of the event content received by the processing lambda, without success – I suspect the incoming request to S3 to create the object may well be completely distinct from the context of the S3 trigger (which fires regardless of where the new object came from). We've also looked for some way to map an Authorization header (or mapped query parameter) to S3 metadata, again, no joy. The AWS S3 integration service is a black box with black ports, not well documented, so I can't tell if there's a way to do this

Worst case we can probably use the RequestId to correlate the ObjectCreated event with some kind of log entry, but that's more fragile than I'd prefer; I'd much rather do it as part of proxy integration configuration if possible.

Any thoughts?

S3 PUT supports a request header `X-Amz-Tagging` with a format `key1=value1&key2=value2&...` (etc.) that will tag the object with the supplied url-encoded (yes, in a header) key-value pairs. If you can map it that way, it might be a solution. You'd have to fetch the object tags. — Michael - sqlbot, Sep 17 '19 at 01:19
That's the missing link i've been searching for. Yes, the method integration config allows one to include a incoming request value (params, headers, context) in a header. So we could map the API key id to a `X-Amz-Tagging` header, which you're saying would in turn be converted to a tag on the S3 object. Just what I need. Thanks. (ps, if you move your comment to an actual answer I can give you credit for the response :-) — Peter Halverson, Sep 18 '19 at 14:27

score 2 · Accepted Answer · answered Sep 18 '19 at 19:29

Unless there's a "gotcha" in API Gateway that makes this not work the way it needs to, you can add tags to an S3 object as you upload it, by injecting the right header into the request.

S3's PUT Object REST method supports a request header, X-Amz-Tagging, which allows you to specify one or more tag keys and values, and you should be able to inject this in the integration request -- though I'm not certain whether API Gateway provides enough functionality to let you do string concatenation and apply the url-encoding this header expects. You potentially won't need to use URL-encoding unless there are characters in the API keys that require it, and there may not be any such characters.

I also discussed this in the answer to Difference between object tags and object metadata?, which makes another important point -- tags may be appropriate for this, but metadata is not. Object metadata is provided to the client every time an object is downloaded, which would potentially reveal those API keys to untrusted parties downloading the object. Tags are not included with a download -- there's a separate subresource used for fetching those. Your Lambda event handler will need to make the call to S3 to fetch the tags, if they're needed in the event handler, or if you just need it archived "in case" then the presence in the tags may be sufficient.

Thanks! See my response below for how to inject an `x-amz-tagging` header into an APIGW request. — Peter Halverson, Sep 20 '19 at 15:21
Nicely done. Feel free to un-accept my answer and accept your own. Note that a fully correct implementation would use [`$.util.urlEncode()`](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html#util-template-reference) to escape any reserved characters the tag key and tag value individually (but not the `=` between them and not the joining `&` if you wanted more than one tag). In your scenario, there likely are no such characters. — Michael - sqlbot, Sep 20 '19 at 16:36
Correction to the above, it is of course `$util.urlEncode()`. — Michael - sqlbot, Sep 21 '19 at 00:08
Correct, urlencoding isn't necessary here, since an api key id is alphanumeric, but I've updated the example anyway, in case someone tries it adapt it without reading the commentary. Thanks again. — Peter Halverson, Sep 21 '19 at 21:22

Peter Halverson · Answer 2 · 2019-09-21T21:21:06.773

Following up on Michael's answer, APIGW does indeed support constructed header mappings, although it's a bit tricky. Turns out you can use a mapping template to set headers and params, not just transform the body. The full explanation is covered in Use a Mapping Template to Override an API's Request and Response Parameters and Status Codes, but, briefly, I created an application/json integration request mapping template with the following content:

#set($tags="submitter_api_key_id=$util.urlEncode($context.identity.apiKeyId)")
#set($context.requestOverride.header.x-amz-tagging = $tags)
$input.json("$")

Note setting the x-amz-tagging header is done as a side-effect of the body transform, which is just a pass-through ($input.json("$") injects the entire body as JSON).

The only other change was to grant tagging permissions to the API execution role, ala

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObjectTagging"
            ],
            "Resource": "arn:aws:s3:::<origin-bucket>/*"
        }
    ]
}

That was it. My POSTed S3 objects are now created with a tag named submitter_api_key_id, which contains the ID of the API key associated with the request. I can then use that key to map to the identity of the actual submitter and enforce our business policies.

Get AWS request API key in S3 bucket trigger?

2 Answers2