AWS Lambda in VPC sometimes doesn't have internet access

Question

I have Lambda which was deployed to VPC.

This deploymens has next configs:

VPC (192.168.0.0/16)
Public Subnet A (192.168.32.0/20) has NAT Gateway and Route 0.0.0.0/0 to Internet Gateway
Private Subnet A (192.168.48.0/20) has Route 0.0.0.0/0 to NAT Gateway
Private Subnet B (192.168.64.0/20)

Lambda has own Securiy Group and references to "Private Subnet A" and "Private Subnet B"

I have strange problem: time to time Lambda doesn't have Internet Access. 3rd party service works normal.

One more strange thing that Lambda gets IP's like 127.0.0.1, 169.254.76.13, 169.254.79.1 instead of IP's from Subnets (192.168.48.0/20 and 192.168.64.0/20).

Error:

Error: connect ETIMEDOUT x.x.x.x:443
at Object._errnoException (util.js:1022:11)
at _exceptionWithHostPort (util.js:1044:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1198:14)

Deployment schema:

Here full CloudFormation template:

    ---
AWSTemplateFormatVersion: '2010-09-09'
Description: 'Base Infrastructure'
Metadata:
  'AWS::CloudFormation::Interface':
    ParameterGroups:
    - Label:
        default: 'VPC Parameters'
      Parameters:
      - VpcId
      - InternetGatewayId
Parameters:
  VpcId:
    Type: String
  InternetGatewayId:
    Type: String
Resources:
  SubnetAPublic:
    Type: 'AWS::EC2::Subnet'
    Properties:
      AvailabilityZone: !Select [0, !GetAZs '']
      CidrBlock: !Sub '192.168.32.0/20'
      MapPublicIpOnLaunch: true
      VpcId: !Sub '${VpcId}'
  SubnetAPrivate:
    Type: 'AWS::EC2::Subnet'
    Properties:
      AvailabilityZone: !Select [0, !GetAZs '']
      CidrBlock: !Sub '192.168.48.0/20'
      VpcId: !Sub '${VpcId}'
  SubnetBPrivate:
    Type: 'AWS::EC2::Subnet'
    Properties:
      AvailabilityZone: !Select [1, !GetAZs '']
      CidrBlock: !Sub '192.168.64.0/20'
      VpcId: !Sub '${VpcId}'
  RouteTablePublic:
    Type: 'AWS::EC2::RouteTable'
    Properties:
      VpcId: !Sub '${VpcId}'
  RouteTablePrivate:
    Type: 'AWS::EC2::RouteTable'
    Properties:
      VpcId: !Sub '${VpcId}'
  RouteTableBPrivate:
    Type: 'AWS::EC2::RouteTable'
    Properties:
      VpcId: !Sub '${VpcId}'
  RouteTableAssociationAPublic:
    Type: 'AWS::EC2::SubnetRouteTableAssociation'
    Properties:
      SubnetId: !Ref SubnetAPublic
      RouteTableId: !Ref RouteTablePublic
  RouteTableAssociationAPrivate:
    Type: 'AWS::EC2::SubnetRouteTableAssociation'
    Properties:
      SubnetId: !Ref SubnetAPrivate
      RouteTableId: !Ref RouteTablePrivate
  RouteTableAssociationBPrivate:
    Type: 'AWS::EC2::SubnetRouteTableAssociation'
    Properties:
      SubnetId: !Ref SubnetBPrivate
      RouteTableId: !Ref RouteTableBPrivate
  EIP:
    Type: 'AWS::EC2::EIP'
    Properties:
      Domain: vpc
  NatGateway:
    Type: 'AWS::EC2::NatGateway'
    Properties:
      AllocationId: !GetAtt 'EIP.AllocationId'
      SubnetId: !Ref SubnetAPublic
  RouteTablePublicInternetRoute:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref RouteTablePrivate
      DestinationCidrBlock: '0.0.0.0/0'
      NatGatewayId: !Ref NatGateway
  RouteTablePublicInternetRoute2:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref RouteTablePublic
      DestinationCidrBlock: '0.0.0.0/0'
      GatewayId: !Sub '${InternetGatewayId}'
  ServerlessSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: SecurityGroup for Serverless Functions
      VpcId: !Sub '${VpcId}'
  ServerlessSecurityGroupIngress:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !Ref ServerlessSecurityGroup
      IpProtocol: -1
      SourceSecurityGroupId: !Ref ServerlessSecurityGroup

Any ideas what I do wrong?

P.S.: I found similar issues AWS VPC Lambda Function keeps losing internet access and Why AWS lambda functions In a VPC sometimes timeout and sometimes work fine?

UPD

Added Route and it works now

  RouteTableBPrivateInternetRoute:
    Type: AWS::EC2::Route
      Properties:
        RouteTableId: !Ref RouteTableBPrivate
        DestinationCidrBlock: '0.0.0.0/0'
        NatGatewayId: !Ref NatGateway

score 1 · Accepted Answer · answered Jul 17 '18 at 12:39

1

For an AWS Lambda function running inside a VPC to be able to access resources outside the VPC (such as the Internet), it must be in a private subnet with a NAT gateway. In your instance Private Subnet A is the only subnet with the appropriate configuration to allow a Lambda function to access the Internet. So you need to edit your Lambda function's configuration to only run in that subnet.

answered Jul 17 '18 at 12:39

Mark B

183,023
24
297
295

As I know Lambda require at least one subnet in each Availability Zone https://docs.aws.amazon.com/lambda/latest/dg/vpc.html#vpc-setup-guidelines – jincod Jul 18 '18 at 05:09
Also I have Route from Private Subnet A (0.0.0.0/0 -> NAT), is it not enough? – jincod Jul 18 '18 at 05:29
1

Route from private subnet A to NAT is enough for the Lambda function to work in private subnet A. You have to have that same route in each subnet you want the Lambda function to have Internet access in. – Mark B Jul 18 '18 at 11:52
If you read the documentation page you linked, it clearly says that only private VPC subnets with a NAT gateway route can give the Lambda function Internet access. It also clearly says that "one subnet in each Availability Zone" is recommend. Not sure how you got "require" out of "recommend" – Mark B Jul 18 '18 at 11:53

score 1 · Answer 2 · answered Jan 23 '20 at 19:04

Have you selected your public subnet as one of the subnets your lambda will run in?

A lambda can only access the internet when it is running in private subnets, so go to your lambda page on the AWS Console, deselect the public subnet and save and the issue shouldn't re-occur.

AWS Lambda in VPC sometimes doesn't have internet access

2 Answers2

Linked