Can not access S3 via VPC endpoint in Lambda

Question

I have a Lambda function in my VPC, and I want to access S3 bucket.

I have setup the S3 VPC endpoint correctly, I think, because I created an EC2 instance in the same subnet and security group as the Lambda function. When I ran a copy of Lambda function code on the EC2 instance, it can correctly showed the S3 file content.

But when I run the code in Lambda, it failed. So, I want to know what is the difference between "run in EC2" and "run in Lambda"? Why did it fail when I ran it in Lambda?

Here is my Lambda function code:

    import boto3
    
    s3 = boto3.client('s3', region_name='ap-northeast-1')
    
    def lambda_handler(event, context):
        bucket = '*xxxxxx*'
        key = 's3-upload.json'
        try:
            response = s3.get_object(Bucket=bucket, Key=key)
            print('--------------------------------------')
            print(response)
            print('--------------------------------------')
            body = response['Body'].read()
            print(body)
            print('--------------------------------------')
            print("CONTENT TYPE: " + response['ContentType'])
            
        except Exception as e:
            print('Error getting object.')
            print(e)
            raise e

John Rotenstein · Accepted Answer · 2019-10-15T05:09:10.710

11

If you want to allow an AWS Lambda to access Amazon S3, use one of these methods:

Do not associate the function to a VPC. Access is then automatic.
~~If the function is attached to a public subnet in the VPC, associate an Elastic IP to the Lambda function's ENI that appears in the VPC~~ (Not recommended)
If the function is attached to a private subnet in the VPC, launch a NAT Gateway in the public subnet and update Route Tables. Traffic will flow to the Internet via the NAT Gateway.
Add an Amazon S3 VPC Endpoint in the VPC and update Route Tables. Traffic will flow through that instead of the Internet Gateway.

edited Oct 15 '19 at 05:09

answered May 26 '18 at 01:29

John Rotenstein

241,921
22
380
470

How do you find out lambda function's ENI ? – Rafaf Tahsin Oct 15 '19 at 04:04
@RafafTahsin You typically do not need to know about the ENI. It is created automatically by Lambda and can also be deleted automatically. The same Lambda function can use multiple ENIs if it scales-out onto many containers. – John Rotenstein Oct 15 '19 at 04:13
Actually I was curious about option 2 you mentioned. I was trying to associate Elastic IP with my lambda function's ENI. But how ? Specially when I don't know the eni :| – Rafaf Tahsin Oct 15 '19 at 04:58
1

Ah! I now do not recommend that option. It works, but it will stop working when other ENIs are added/removed. If you wish to play around with it, you can look at the **Network Interfaces** in the EC2 console and figure out which one was created by Lambda. – John Rotenstein Oct 15 '19 at 05:10
Last one worked for me, followed instructions here: https://stackoverflow.com/a/57742313/1327815 – Dfranc3373 Jun 24 '20 at 05:02
For me, everything was setup correctly (Endpoint, Nat, Gateway, etc) but it still wouldnt work. Then I relalised the security group containing the Lambda function had a single Outbound Rule set towards the default security group of the VPC... replaced it with `0.0.0.0/0` as destination and it worked – Guillaume Jan 26 '21 at 08:23

score 0 · Answer 2 · answered May 25 '18 at 09:04

0

Even though they're in the same VPC, EC2 and Lambda are still different environments within AWS. Being able to run your code in one and not the other implies that your code is fine and works, so it's likely to be a configuration issue with AWS.

Have you checked the service/execution role that the lambda is using?

You need to ensure that the IAM role that it's using is allowed the correct level of S3 access.

This documentation on execution roles for lambda might provide a useful jumping off point: https://docs.aws.amazon.com/lambda/latest/dg/intro-permission-model.html#lambda-intro-execution-role

An IAM policy like this would give whatever execution role you use read-only access to all your S3 buckets, and happens to be one of the AWS managed policies.

{
"Version": "2012-10-17",
"Statement": [
    {
        "Effect": "Allow",
        "Action": [
            "s3:Get*",
            "s3:List*"
        ],
        "Resource": "*"
    }
]

}

answered May 25 '18 at 09:04

Robin

300
1
4
11

Thanks，my role have the "AmazonS3FullAccess" policy. – fish May 25 '18 at 09:13
Huh! That's strange then, running the same thing in my AWS account gives no errors. Do you have any error messages you can add to the original question? – Robin May 25 '18 at 09:29
@YOUNG did you attached the vpc endpoint entry to the route table to which your subnet is associated. Also did you assign the role to lambda and what error did you get, is there permission issue or s3 request is timing out. – Mahtab Alam May 25 '18 at 10:46
@Robin The error message is just “time out”. My role have full s3 access policy. And also I add the vpc endpoint rule to my subnet route table（or my ec2 code access s3 too）. – fish May 26 '18 at 13:23

score 0 · Answer 3 · answered May 28 '18 at 02:41

Thanks everyone! I found the reason.

My Lambda have two subnets, private_sn_1 and private_sn_2,

private_sn_1 have correctly set the vpc endpoint route table,

but the private_sn_2 set a wrong route table,

and my ec2 created in private_sn_1 so it can access the vpc endpoint.

In normal, Lambda will run randomly in private_sn_1 or private_sn_2,

but in my case it always run in private_sn_2(I don't know why),

so when I fixed the private_sn_2 route table,

everything is right.

Oleksii Donoha · Answer 4 · 2019-10-24T10:05:24.540

In addition to all said above, it is also possible that VPC Endpoint policy can be prohibitive and not allowing traffic to/from S3 through. Make sure you allow traffic through endpoint by using "Full access" policy.

Edit: here's related bit of documentation:

Your policy must contain a Principal element. For gateway endpoints only, you cannot limit the principal to a specific IAM role or user. Specify "*" to grant access to all IAM roles and users. Additionally, for gateway endpoints only, if you specify the principal in the format "AWS":"AWS-account-ID" or "AWS":"arn:aws:iam::AWS-account-ID:root", access is granted to the AWS account root user only, and not all IAM users and roles for the account.

So for S3 endpoints to work you need to specify '*' as a principal in general case

Can not access S3 via VPC endpoint in Lambda

4 Answers4

Linked

Related