SQS Timeout from Lambda within VPC

Question

I have a Lambda that needs to be on a VPC to talk to protected resources like RDS and AWSDocumentDB. It also needs to be able to see the outside world for some calls to 3rd party APIs. To do this I used the VPC wizard to create a VPC that had both public and private subnets. The wizard also created and attached an Internet Gateway.

After this I attached my Lambda, RDS instance and DocumentDb cluster to the VPC. Since then, however I have been unable to talk to my SQS queues from within my lambda using the NodeJS aws-sdk.

I want to add that I have read and implemented some points from: AWS Lambda: Unable to access SQS Queue from a Lambda function with VPC access however I am still unable to connect.

Here is what I have:

1. VPC:

   • VPC has public and private subnets, and an IG Gateway. I used the wizard to create it. I don't understand much of the underpinnings here.
   • VPC Config (sorry it's a link, it won't let me embed yet.)
   • CIDR's- the wizard created all but the last block. I am unsure if I did this right or if it even matters as the wizard made me create at least one and I did that to avoid IP overlap.
   • As this is a dev/prototype project the security group attached to the VPC is 'wide open'. All inbound and outbound is allowed.
   • Let me know what other VPC config to show as I'm not sure what's useful

2. Service Endpoint:

   • I tried creating a service endpoint for SQS per the article linked above, here's what I have: endpoint config
   • I'll talk more about how I'm consuming this in the Lambda section.
   • The endpoint is attached to the VPC

3. Lambda:

   • I have the Lambda attached to the VPC as shown here.
   • This allows me to talk to public 3rd party api's and my protected resources. It was my hope that having a wide open SG would still allow my lambda to talk to SQS but it keeps timing out.
   • I am unsure which url to use in my Lambda for my endpoint. The example here: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-sending-messages-from-vpc.html looks like it still uses the regional endpoint.

4. Code

   • Here is what the SQS invocation looks like from within my code:

     • const {SQS} = require('aws-sdk');
       
       // Constructor Init
       const sqs = new SQS({
          apiVersion: '2012-11-05', 
          endpoint: 'https://sqs.us-west-2.amazonaws.com', // not sure if this is 'invoking' the vpc endpoint or not
          region: 'us-west-2'
       });
       
       // Send message
       await sqs.sendMessage({
          MessageBody: 'Test body',
          QueueUrl: 'https://sqs.us-west-2.amazonaws.com/<rest of URI>',
          MessageAttributes: {...someAttrs}
       }).promise();
       
       
       
       

Appreciate any help, please let me know what other information I can provide.

Thanks!

** Edit **

I should also mention that to circumvent this whole issue I started to go down the road of using SQS as a Lambda Destination. While this does inject messages into the target queue, it likely will not scale with my use case. I can elaborate on that further if needed as it isn't totally pertinent to the actual question.

** EDIT 8/31/20 **

Thanks for all the responses, it was a great help and got me to a resolution. I will say that to anyone else who finds this post is to first watch:

https://www.youtube.com/watch?v=JcRKdEP94jM

This is something that I wish I found before I started all of this because, while it's specifically targets to giving lambdas internet access, it goes through the process of mapping IG's and Nats to subnets which is really where I was mis-configuring my vpc. With this video I went and re-created my entire VPC and it is so much cleaner and easier to connect the dots. 10/10 recommend.

Thanks again!

Answer 1

The AWS Lambda function should be attached to a private subnet in the VPC.

Amazon SQS lives on the Internet, so the Lambda function needs a way to access the SQS endpoint.

Option 1: VPC Endpoint

A VPC Endpoint for Amazon SQS can provide a direct link between a VPC and the SQS endpoint, without requiring Internet access. Once created, the VPC will automatically send SQS requests across the VPC Endpoint. This is a nice, simple option.

Option 2: NAT Gateway

If you have additional resources in the private subnet(s) that require Internet access, you could consider placing a NAT Gateway in a public subnet. The private subnet(s) will need a Route Table entry that directs Internet-bound traffic (0.0.0.0/0) to the NAT Gateway.

Additional charges apply.

Option 3: Lambda Destination

If you have the Lambda function successfully communicating the SQS via a Lambda Destination, then this sounds like an excellent option! I haven't tried it, but it sounds like the Lambda service is responsible for sending the output directly to SQS, without having to traverse the VPC or Internet. If this is working, then I highly recommend to keep using it. I do not perceive any scaling problems using this method. However, please note that it only works for asynchronous calls to Lambda and will not work, for example, when invoking the function synchronously or by pressing the Test button.

Answer 2

My hunch on this one is that there is a missing rule somewhere in your network configuration - Packets getting dropped either toward your SQS, or on their way back (both need thought through hop by hop).

The three things that come to mind:

1. Routing: Ensure that the subnets and routing table(s) have appropriate routes to get packets back to your private subnet from the one where SQS endpoint is.
2. Security Groups - Look carefully at each SG involved. The SQS may be in a SG that restricts access to it, for example.
3. Network ACLs - these are stateless, so you'll need to ensure both sides are open, and remember that most of the time, there will be random port numbers involved going back to the requester.

Good luck!