0

I have a very simple python function in a lambda which runs fine if I leave VPC disabled.

import json
import boto3
import botocore


def lambda_handler(event, context):
    s3 = boto3.client('s3', 'us-east-1',
                      config=botocore.config.Config(s3={'addressing_style': 'path'}))
    keys = []
    resp = s3.list_objects_v2(Bucket='[BUCKET_NAME]')
    for obj in resp['Contents']:
        print(obj['Key'])

    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

When VPC is enabled the S3 connection continually times out.

I have gone through many documents, tutorials, forum threads and stack overflow postings, but none of them have helped me.

My network ACL has 0.0.0.0/0 mappings for ports 80, 443 and 5439 (Redshift).

My one and only security group has 0.0.0.0/0 mappings for ports 80, 443 and 5439 (Redshift).

I have only one VPC configured.

I have 1 NAT Gateway configured.

I have 1 Internet Gateway configured.

I have 6 subnets in the VPC:

  • Subnets A and B point to the main route table.
  • Subnets C and D point to the 'lambda_rt_table_gateway' route table.
  • Subnets E and F point to the 'lambda_rt_table_nat' route table.

I have 2 endpoints in the VPC:

  • Endpoint VPCE-A is defined for service 'com.amazonaws.us-east-1.s3' and is mapped to all 3 route tables.
  • Endpoint VPCE-B is defined for service 'com.amazonaws.us-east-1.dynamodb' and is mapped to all 3 route tables.

Finally, I have 3 Route Tables:

  • The main route table has the following routes:

  • 172.31.0.0/1 --> local

  • pl-02cd2c6b (com.amazonaws.us-east-1.dynamodb, 52.94.0.0/22, 52.119.224.0/20) --> vpce-07a6eb423bbbea151

  • pl-63a5400a (com.amazonaws.us-east-1.s3, 54.231.0.0/17, 52.216.0.0/15) --> vpce-0fd10c890bb176b5a

  • 0.0.0.0/0 --> igw-04b6aa7c

  • The 'lambda_rt_table_gateway' route table has identical routes as the main.

  • The 'lambda_rt_table_nat' route table has identical routes as well except for the last entry, it is

  • 0.0.0.0/0 --> nat-0a5c0a76e3c12c42f

I am pretty sure it is something simple I'm missing. Please help.

Thanks a lot.

MatthewMartin
  • 32,326
  • 33
  • 105
  • 164
Garet Jax
  • 1,091
  • 3
  • 17
  • 37
  • Question: How much of the above have you created to specifically solve this situation vs having it for other purposes? For example, is something else using the NAT Gateway or VPC Endpoint? – John Rotenstein Feb 06 '19 at 03:42
  • Why are you using `addressing_style:path`? – John Rotenstein Feb 06 '19 at 03:56
  • All this configuration was done attempting to solve this issue. It can all be removed if that is the right approach. This is my test environment and I want to figure out the right way to solve this problem. – Garet Jax Feb 06 '19 at 04:24
  • I am using the addressing_style:path because of this SO thread: https://stackoverflow.com/questions/39779962/access-aws-s3-from-lambda-within-vpc/39797802 – Garet Jax Feb 06 '19 at 04:24
  • 1
    Probably the worst possible provisioning error I see people make is changing Network ACLs before ensuring that the setup otherwise works as expected. I see no allowance mentioned for ephemeral ports. Set the Network ACLs back to normal (allowing all in *both* directions) before proceeding. – Michael - sqlbot Feb 06 '19 at 12:13
  • I started from scratch and created two additional VPCs - one with one public subnet and one with a private and a public subnet. I am working through getting them to work using the VPC endpoints. My first simple test was very successful. – Garet Jax Feb 07 '19 at 03:42

1 Answers1

3

You have a lot of stuff configured! I'm not sure how much of it is part of wanting to get this specific situation fixed, or whether you have other needs for things like the NAT Gateway, VPC Endpoints, etc.

The simplest setup to enable a VPC-connected Lambda function to call out to the Internet (eg to make an API call to Amazon S3) would be:

  • Add a NAT Gateway to a Public subnet
  • Attach the Lambda function to a Private subnet
  • Set routing on the private subnet to use the NAT Gateway for 0.0.0.0/0

That is sufficient for VPC-attached Lambda functions to reach the Internet.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • Thanks - I am not trying to get Lambda to reach the internet. I am trying to do what is in this blog post: https://aws.amazon.com/blogs/aws/new-vpc-endpoints-for-dynamodb/ – Garet Jax Feb 06 '19 at 11:31
  • I am effectively trying to keep all communication inside of AWS using VPC end points. – Garet Jax Feb 06 '19 at 11:34
  • Is there any reason for locking it down like this? Understandable if it is a Dev/Test VPC connected to your on-premises network via Direct Connect where you want no internet connectivity, but if you're going to use an Internet Gateway, then you need a good reason to use VPC Endpoints rather than normal endpoints. – John Rotenstein Feb 06 '19 at 19:47
  • Hmmm. Maybe I am missing something, but it seems to me that VPC endpoints should be the default choice. They are more secure since they keep all communication inside the VPC. – Garet Jax Feb 07 '19 at 03:55
  • I guess it depends on how extreme you wish to go for security. Request to AWS APIs are made via HTTPS, so the message content in transit is secure. – John Rotenstein Feb 07 '19 at 04:25
  • I was able to get a Lambda functioning against both S3 and Dynamo using the VPC endpoint and without having to create a private subnet. However since there is no VPC endpoint for Redshift, I had to use the above approach. Thanks for your time. – Garet Jax Feb 07 '19 at 05:14