21

I have a Lambda function that needs to read messages from an SQS queue using it's URL. Then it needs to insert that data to Cassandra running on a server inside a VPC.

I am able to access the Cassandra server from my Lambda function, using it's private IP and configuring the security groups correctly.

However, I am not able to read messages from the SQS Queue. When I change the configuration of Lambda function to No VPC, then I am able to read the messages from the SQS Queue. However, with VPC settings, it just times out.

How can I overcome this ? I have checked the security group of my Lambda function has full outbound access to all IP addresses.

Mandeep Singh
  • 7,674
  • 19
  • 62
  • 104

5 Answers5

20

At the end of 2018, AWS announced support for SQS endpoints which provide

connectivity to Amazon SQS without requiring an internet gateway, network address translation (NAT) instance, or VPN connection.

There is a tutorial for Sending a Message to an Amazon SQS Queue from Amazon Virtual Private Cloud

See also the SQS VPC Endpoints Documentation for more information.

Its important to note that if you want to access SQS within the Lambda VPC there are a couple other things you need to do:

  • Make sure to specify the SQS region in your code. For example, I had to set my endpoint_url to "https://sqs.us-west-2.amazonaws.com"
  • Make sure that you have attached a "wide open" security group to the SQS VPC Interface, otherwise SQS will not work.
  • Make sure that your subnets in your Lambda VPC match what you have set up for your SQS VPC Interface.
jasonrhaas
  • 1,083
  • 9
  • 11
Onema
  • 7,331
  • 12
  • 66
  • 102
  • 3
    Can this be leveraged to have SQS trigger a Lambda function residing inside a VPC as well? – Rajeev Atmakuri Jan 28 '19 at 10:24
  • @RajeevAtmakuri, yes. I used it for that and it worked as expected. Make sure to follow the setup guide and add endpoints for any other services you may be using in the function. – Onema Jan 29 '19 at 00:18
  • 1
    Would love to know if anybody got this working from inside a nodejs handler (meaning using the SQS JS SDK sendMessage method). I can't find anywhere that documents support for the endpoint-url parameter that is described in the CLI example in the tutorial here. In fact the SQS API doesn't even describe it. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-sending-messages-from-vpc.html#sqs-vpc-tutorial-publish – Taylor Apr 13 '19 at 02:12
  • Never mind. Apparently you don't need that param at all from the JS. It "just works". :) – Taylor Apr 13 '19 at 02:28
  • 1
    `Make sure that you have attached a "wide open" security group to the SQS VPC Interface, otherwise SQS will not work.` If you attach a wide open outbound security group policy, then you dont need a VPC end point with SQS service. You Lambda can directly invoke publicly available SQS endpoint directly. – Aditya Verma Sep 20 '19 at 13:39
  • Thank you, worked Step 4 for me....Step 4: Create an Amazon VPC endpoint for Amazon SQS - https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-sending-messages-from-vpc.html – Aditya Y Dec 30 '22 at 14:14
13

Some services (e.g. S3) are offering VPC endpoints to solve this particular problem but SQS is not one of them. I think the only real solution to this problem is to run a NAT inside your VPC so the network traffic from the Lambda function can be routed to the outside world.

garnaat
  • 44,310
  • 7
  • 123
  • 103
  • I have no clue how to do that. could you point me in the right direction ? – Mandeep Singh Feb 16 '16 at 19:50
  • It has gotten easier. AWS now has a managed NAT service. Here's a blog post describing it: https://aws.amazon.com/blogs/aws/new-managed-nat-network-address-translation-gateway-for-aws/ – garnaat Feb 16 '16 at 20:04
  • 1
    I tried adding an entry to the route table for destination `0.0.0.0/0` with the target as the id of the newly created NAT gateway. However, there was already an entry for Internet gateway with the same destination. Is it safe to replace that ? – Mandeep Singh Feb 17 '16 at 09:16
  • 1
    Ok I think it does not work that way. Once I replaced the entry, I could no longer SSH to my instance. How to go about that ? – Mandeep Singh Feb 17 '16 at 09:21
  • 18
    Am I the only one that reads this kind of "just create a NAT and edit your routing tables and setup your VPC endpoints and ..." and thinks it's too much? I'm pretty sure if AWS offered peanut butter and jelly sandwiches, it would require setting a permission policy on the peanut butter jar and creating special separate subnets for each slice of bread to reside in. – Micah Jul 14 '16 at 20:14
  • 8
    This answer is no longer correct. AWS offers VPC endpoints for SQS. – jasonrhaas Apr 14 '19 at 17:21
  • What is a VPC endpoint for SQS? What is the heart of the problem here... is SQS not in the VPC? – Jwan622 Oct 02 '19 at 00:51
  • 1
    @Jwan622 no. You can add services like Lambda, EC2 and RDS into your own VPC, but not SQS nor S3. Usually SQS and S3 would require your EC2 or Lambda to have internet access so you can reach them, which can be tricky. A VPC Endpoint allows AWS Services (EC2, Lambda) in a VPC to access other AWS Services like SQS without leaving AWS Network, thus not requiring internet access. – Alisson Reinaldo Silva Jul 21 '20 at 18:54
7

I ran into the same kind of problem when I was running lambda function with access to elasticache on the VPC. While the function was configured to run in the VPC, I wasnt able to talk to any other service (specifically codedeploy for me).

As @garnaat pointed out NAT seems to be the only way to go about solving this problem for services without VPC endpoints.

And like you pointed out, I also ran into the same trouble where I could'nt SSH into the machine(s) once I replaced the entry with the IGW in the route table. Seems like detaching the IGW starves the VPC of either the incoming traffic (mostly) or the outgoing traffic from or to the internet respectively. So here's what I did and it worked for me:

Create a new Subnet within the VPC Now, when lambda runs, make sure lambda operates from this subnet. You can do this by using aws-cli like so:


 aws lambda update-function-configuration --function-name your-function-name --vpc-config SubnetIds="subnet-id-of-created-subnet",SecurityGroupIds="sg-1","sg-2"

Make sure you add all the security groups whose inbound and outbound traffic rules apply for your lambda function.

Next, go to Route Tables in the VPC console and create a new route table.

enter image description here

Here is where you add the NAT gateway to the target.

finally go to the Subnet Associations tab in the new route table and add the newly created subnet there.

Thats all this should get it working . Mind you, please treat this as only a workaround. I haven't done much digging and I have a very limited idea on how things get resolved internally while doing this. This might not be an ideal solution.

The ideal solution seems to be to design the VPC before hand. Use subnets to isolate resources/instances that need internet access and that dont(private and public subnets) and place appropriate gateways where needed.( so that you may not have to create a seperate subnet for this purpose later). Thanks

Avinragh
  • 248
  • 3
  • 10
3

I was unable to get either of the other two answers to this question to work. Perhaps this is due to one or more mistakes on my part. Regardless, I did find a workaround that I wanted to share, in case I'm not alone with this problem.

Solution: I created two Lambda functions. The first Lambda function runs inside my VPC and performs the desired work (in mandeep_m91's case, that's a data insertion to Cassandra; in my case is was accessing an RDS instance). The second Lambda function lives outside the VPC, so I could hook it up to the SQS queue. I then had the second Lambda function call the first, using the information found this this StackOverflow Q&A answer. Note, the linked question has both node.js and Python examples in the answers.

This will effectively double the cost of making a function call, since each call results in two function executions. However, for my situation, the volume is so low it won't make a real difference.

Community
  • 1
  • 1
Russell Thackston
  • 191
  • 6
1

To clarify a point above about a "wide open" security group, the group set on the endpoint needs to allow inbound access to SQS from your lambda function.

I created a security group for my endpoint that only opened 443 to my lambda's security group.

James
  • 51
  • 1
  • 4