API Gateway+Lambda+VPC timeout issue

Question

Good morning, Could you please help us with next problem: I have an API Gateway + Java Lambda Handler. this Lambda uses httpconnection to get some Internet REST API.

when we use this Lambda without VPC it works fine. but when we are using VPC with configured internet access - sometimes Lambda fails with timeout errors. it fails in 20% of all requests (80% requests works fine) with next errors at log.

REPORT RequestId: 16214561-b09a-11e6-a762-7546f12e61bd Duration: 15000.26 ms Billed Duration: 15000 ms Memory Size: 512 MB Max Memory Used: 47 MB

09:57:49
2016-11-22T09:57:49.245Z 16214561-b09a-11e6-a762-7546f12e61bd Task timed out after 15.00 seconds

According to my logs lambda cannot send GET request. I'm not sure where the problem at. Is this Lambda issue, VPC issue or some cofiguration issue. Also I did try many different REST Api endpoints, so it's definetly not an endpoint issue.

Appreciate any help.

score 6 · Answer 1 · answered Nov 22 '16 at 19:11

6

When you place a Lambda function inside your VPC it will not have access to anything outside the VPC. To enable your Lambda function to access resources outside the VPC you have to add a NAT Gateway to your VPC.

answered Nov 22 '16 at 19:11

Mark B

183,023
24
297
295

4

...and choose only private subnets (with a defaut route to the NAT box) for assignment the Lambda function, I'm thinking, since the Lambda ENIs would not be assigned public IPs even if provisioned on a public subnet. – Michael - sqlbot Nov 23 '16 at 02:18
@Michael-sqlbot Thank you very much, you are right that was the problem – Svyatoslav Babych Nov 23 '16 at 22:03
I have similar issue. and configured private Subnet with NAT Gateway, even so, still have no access to Internet. Any help? – Kostanos Feb 13 '17 at 16:43
@Kostanos Does an EC2 instance in that private subnet have Internet access? You might have your routing table setup incorrectly. I would suggest asking a new question on here instead of tagging onto this one. – Mark B Feb 13 '17 at 16:45
Thank you @mark-b I already resolved it. My issue was that I created NAT gateway inside of private subnet, when I should do it inside public subnet. – Kostanos Feb 13 '17 at 20:52

score 3 · Answer 2 · answered Nov 23 '16 at 22:06

3

The problem is solved.

Lambda VPC configuration had public subnet attached.

Thanks to @Michael-sqlbot

answered Nov 23 '16 at 22:06

Svyatoslav Babych

61
1
4

score 0 · Answer 3 · answered Jan 30 '17 at 23:55

I had pretty much the same issue a few months ago, and here is my solution:

Assuming you set up your Lambda manually, in the Configuration -> Advanced settings you will find the VPC and then choose subnet and security groups.

The Subnet you selected should be in the same subnet with other services the lambda function invokes. In your case, your lambda service uses httpconnection to Internet rest API, that's fine, but you may need DB connection with RDS or triggered by SQS or SNS. So make sure the subnet is correct.

The Security Groups is more important. Again, in your case, you need the access to Internet, so ensure the security group's outbound rules has external connections. Normally, I give all ports and all destination available for simplicity, and of course, you can limit to use port 80 and the API's IP address you need.

score -1 · Answer 4 · answered Aug 04 '21 at 06:13

Since the executor is "locked" behind a VPC - all internet communications are blocked.

That results in any http(s) calls to be timed out as they request packet never gets to the destination.

That is why all actions done by aws-sdk result in a timeout.

Please refer to https://stackoverflow.com/a/39206646

SkyWalker · Answer 5 · 2016-11-22T14:59:48.963

From your log,

Billed Duration: 15000 ms

Memory Size: 512 MB

Max Memory Used: 47 MB

Solution:

It is timeout issue. You need to increase the execution time 15 seconds to 30 seconds or more if necessary.

In some cases, you also need to increase the memory size. It may also make effect. But I think time is the main fact for you, not memory size.

For timing issue and testing issue, you can go through the followings:

Q: How long can an AWS Lambda function execute?

Soluiton: All calls made to AWS Lambda must complete execution within 300 seconds. The default timeout is 3 seconds, but you can set the timeout to any value between 1 and 300 seconds.

To determine why your Lambda function is not working as expected:

You can test your code locally as you would any other Node.js function, or you can test it within the Lambda console using the console's test invoke functionality, or you can use the AWS CLI Invoke command. Each time the code is executed in response to an event, it writes a log entry into the log group associated with a Lambda function, which is /aws/lambda/.

If you see a timeout exceeded error in the log, your timeout setting exceeds the run time of your function code. This may be because the timeout is too low, or the code is taking too long to execute.

For solution:

Test your code with different memory settings.

If your code is taking too long to execute, it could be that it does not have enough compute resources to execute its logic. Try increasing the memory allocated to your function and testing the code again, using the Lambda console's test invoke functionality. You can see the memory used, code execution time, and memory allocated in the function log entries. Changing the memory setting can change how you are charged for duration. For information about pricing, see AWS Lambda.

Resource Link:

Troubleshooting and Monitoring AWS Lambda Functions with Amazon CloudWatch

For testing, a full code example is given here: http://qiita.com/c9katayama/items/b9a30cdfaaa91cba23ad

This would be a good answer, but I'm afraid it's not really addressing the likely issues in *this* question. If they're getting a timeout it may simply be because they're hitting a situation with an outbound connection (to whatever it may be) that is initiated by the Lambda function itself, never succeeding due to a misconfiguration that is specific to the VPC, since OP says it works when deployed non-VPC. I do believe you should reconsider whether you're addressing the likely crux of the issue. There is, however, no reason to believe a memory issue exists. — Michael - sqlbot, Nov 23 '16 at 02:12