Node Lambda AWS TimeoutError: Socket timed out without establishing a connection to cloudformation

Question

I am running a Node(12.x) Lambda in AWS. The purpose of this lambda is to interact with Cloudformation stacks, and I'm doing that via the aws-sdk. When testing this lambda locally using lambda-local, it executes successfully and the stack can be seen in CREATING state in AWS console. However, when I push and run this lambda in AWS, it fails after 15 seconds, and I get this error:

{"errorType":"TimeoutError","errorMessage":"Socket timed out without establishing a connection","code":"TimeoutError","message":"Socket timed out without establishing a connection","time":"2020-06-29T03:10:27.668Z","region":"us-east-1","hostname":"cloudformation.us-east-1.amazonaws.com","retryable":true,"stack":["TimeoutError: Socket timed out without establishing a connection","    at Timeout.connectTimeout [as _onTimeout] (/var/task/node_modules/aws-sdk/lib/http/node.js:69:15)","    at listOnTimeout (internal/timers.js:549:17)","    at processTimers (internal/timers.js:492:7)"]}

This lead me to investigate the lambda timeout and the possible configuration changes I could make found in https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/ and https://aws.amazon.com/premiumsupport/knowledge-center/lambda-vpc-troubleshoot-timeout/ but nothing worked.

I found a couple of similar issues such as AWS Lambda: Task timed out which include possible suggestions such as lambda timeout and lambda memory issues, but Ive set mine to 30 seconds and the logs show max memory used is 88MB out of possible 128MB, but I tried with an increase anyway, and no luck.

The curious part is that it fails without establishing a connection to hostname cloudformation.us-east-1.amazonaws.com. How is that possible when the role assigned to the lambda has full Cloudformation privileges? I'm completely out of ideas so any help would be greatly appreciated. Heres my code:

TEST EVENT:

{
  "stackName": "mySuccessfulStack",
  "app": "test"
}

Function my handler calls (createStack):

const AWS = require('aws-sdk');

const templates = {
    "test": {
      TemplateURL: "https://<bucket>.s3.amazonaws.com/<path_to_file>/test.template",
      Capabilities: ["CAPABILITY_IAM"],
      Parameters: {
        "HostingBucket": "test-hosting-bucket"
      }
    }
}

async function createStack(event) {
  AWS.config.update({
    maxRetries: 2,
    httpOptions: {
      timeout: 30000,
      connectTimeout: 5000
    }
  });
  const cloudformation = new AWS.CloudFormation();
  const { app, stackName } = event;
  let stackParams = templates[app];
  stackParams['StackName'] = app + "-" + stackName;
  let formattedTemplateParams = [];
  for (let [key, value] of Object.entries(stackParams.Parameters)) {
    formattedTemplateParams.push({"ParameterKey":key, "ParameterValue": value})
  }
  stackParams['Parameters'] = formattedTemplateParams;
  const result = await cloudformation.createStack(stackParams).promise();
  return result;
}

Yes @Marcin it is indeed in a VPC. I tried to tackle that by assigning the lambda to a subnet that has an internet gateway assigned, but that didnt work. What are you thinking? — johnny_mac, Jun 29 '20 at 04:11

score 4 · Accepted Answer · answered Jun 29 '20 at 04:14

Lambda function in a VPC does not public IP address nor internet access. From docs:

Connect your function to private subnets to access private resources. If your function needs internet access, use NAT. Connecting a function to a public subnet does not give it internet access or a public IP address.

There are two common solutions for that:

place lambda function in a private subnet and setup NAT gateway in public subnet. Then set route table from private subnet to the NAT device. This will enable the lambda to access the internet and subsequently CloudFormation service.
setup a VPC interface endpoint for CloudFormation. This will allow your lambda function in private subnet to access CloudFormation without the internet.

Thanks @Marcin I had suspected something like this from the beginning but implemented it wrong and went down another path. Thanks for bringing me back to it with clarification — johnny_mac, Jun 29 '20 at 05:28

Node Lambda AWS TimeoutError: Socket timed out without establishing a connection to cloudformation

1 Answers1