504 timeout accessing S3 from Lambda with boto3

Question

This is not an issue with the size of the files, the files are very small. It works perfectly where running locally, but not in the Lambda.

I attached a bucket-access policy to Lambda role mylambdarole:

which looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket",
                "arn:aws:s3:::my-bucket/*"
            ]
        }
    ]
}

I also added a permission policy to my-bucket:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Lambda permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::XXX843121XXX:role/mylambdarole"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::my-bucket",
                "arn:aws:s3:::my-bucket/*"
            ]
        }
    ]
}

Lambda is in a VPC, but I think this shouldn't be a problem as I have a route to internet gateway:

The endpoint uses boto3, which for example loads data from the bucket:

def get_user_data_from_bucket(user_id: str):
    s3_client = boto3.client("s3", aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY, region_name=REGION)
    try:
        data = s3_client.get_object(Bucket=BUCKET_NAME, Key=f"{user_id}/user_data.json")
        return json.loads(data["Body"].read())
    except ClientError as e:
        if e.response["Error"]["Code"] == "NoSuchKey":
            return {}
        raise e

I also tried adding config=Config(s3={'addressing_style':'path'}) to the client, but it didn't solve the issue.

As I said, it works when running the server locally. However, when the endpoint for downloading data is called, there is no error response, only after some time I get 504: Endpoint request timed out. Accessing RDS from the same Lambda function works just fine.

Why are you using `aws_access_key_id` & `aws_secret_access_key` when using a Lambda execution role? — Ermiya Eskandary, Mar 01 '22 at 12:42
Well I need it to use it locally and I assume the lambda just sets those environment variables. — KarloSpacapan, Mar 01 '22 at 12:48

jarmod · Accepted Answer · 2022-03-01T15:50:02.567

2

You configured your Lambda function to connect to a VPC public subnet. Lambda functions connected to a public subnet cannot access the internet. Even though it routes to the IGW, the Lambda does not have a public IP, and so its outbound packets are dropped at the IGW. More details here.

Re-configure the Lambda function to connect to a private subnet.

Also, there's no need for that S3 bucket policy (use the IAM role) or the manual credentials (use the IAM role).

edited Mar 01 '22 at 15:50

answered Mar 01 '22 at 14:43

jarmod

71,565
16
115
122

Thank you. Based on the answer you linked I looked at VPC endpoints and followed https://stackoverflow.com/a/57742313/13504606, which solved the issue. – KarloSpacapan Mar 03 '22 at 08:52

504 timeout accessing S3 from Lambda with boto3

1 Answers1