3

I set up a very simple Lambda for sending a message to a SQS. Lambda is in a VPC, with two public subnets (I do not fully understand aws networking, I've just seen that the routing table connected to subnets have 0.0.0.0/0 as one the routes, connected to Internet Gateway) and a security group. I've already double-checked Permissions and they work properly, cause if I remove VPC settings on Lambda it works.

I tried to create an Endpoint as suggested in the article: SQS sending from VPC, but the Lambda timed out.

As suggested in SO solution link, I tried to add the endpoint_url in the client, also that's not working.

Lambda code is the following:

#Testing SQS push message.
import botocore
import boto3

def main(event, context):
    session = boto3.Session()

    sqs_client = session.client(
        service_name='sqs',
        endpoint_url='https://sqs.eu-west-1.amazonaws.com',
    )

    sqs_client.send_message(
        QueueUrl='https://sqs.eu-west-1.amazonaws.com/***********/tutorial-queue-test',
        MessageBody='msg sent from '
    )

    return {}

Resuming my setup I have:

  • Lambda inside a VPC, 2 subnets(public), 1 security group.
  • SQS
  • SQS Endpoint inside the VPC

I cannot keep the Lambda outside the VPC, cause I'll need to use a EFS, that I will integrate in the Lambda.

SOLUTION: Afterall I succeded to launch correctly the Lambda, I guess it was a mix of bad security group rules, both for Lambda and Endpoint, and VPC private DNS name disabled. Thanks everyone for the support.

Just for readability purposes I summerise the main solutions that brought me to successfully launch Lambda:

  • Add a Security Group for Lambda, which has INBOUND RULE(Protocol:All TCP, Ports:0 - 65535, Source:0.0.0.0/0) and OUTBOUND RULE(Protocol:All, Ports:All, Destination:0.0.0.0/0)
  • Add a Security Group for Endpoint, which has INBOUND RULE(IP version: –, Type:All TCP, Protocol:TCP, Ports:0 - 65535, Source: <INSERT_LAMBDA_SECURITY_GROUP>) and OUTBOUND RULE(IP version: IPv4, Type:All traffic, Protocol:All, Ports:All, Destination: 0.0.0.0/0).
  • From Endpoints, select the current Endpoint->Actions->Modify private DNS name-> Enable private DNS names.
Stefano Zimmitti
  • 31
  • 1
  • 1
  • 4
  • Please post the details of the security group assigned to the SQS endpoint. Also, please verify you have enabled DNS support on your VPC. – Mark B Mar 24 '22 at 13:22
  • In-bound rules: (sg-08XX75XXX2d0fXX19 Custom TCP 0 0.0.0.0/0), (sg-0bXXX0dcb4ac1XX24 Custom TCP 2049 0.0.0.0/0). Outbound rules: (sg-08XX75XXX2d0fXX19 All All 0.0.0.0/0) (sg-0bXXX0dcb4ac1XX24 All All 0.0.0.0/0). DNS hostnames and DNS resolution are enabled. – Stefano Zimmitti Mar 24 '22 at 13:32
  • In your question you said there is only 1 security group. In your comment just now you listed 2 security group IDs. I take it one of those is assigned to the Lambda function? – Mark B Mar 24 '22 at 13:37
  • Yeah sorry I just added after I posted the the question, is that a problem having two security groups assigned? – Stefano Zimmitti Mar 24 '22 at 13:38
  • Having separate security groups for Lambda and the Endpoint is the correct configuration, I'm just trying to make sure you have posted all the relevant info so we can figure out your issue. – Mark B Mar 24 '22 at 13:40
  • I just dropped one, now I've only one security group, so these are the following rules: INBOUND(sg-08XX75XXX2d0fXX19 Custom TCP 0 0.0.0.0/0) and OUTBOUND(sg-08427XXX02dXX5019 All All 0.0.0.0/0) – Stefano Zimmitti Mar 24 '22 at 13:41
  • The security group assigned to the Lambda function needs to allow all outbound traffic, on any TCP port. The security group assigned to the SQS Endpoint needs to allow all inbound traffic, from the Lambda function's security group, on port 443. You have a rule that only allows TCP port `0` right now, which is incorrect. – Mark B Mar 24 '22 at 13:43
  • What "type" should I choose for the Inbound Rule of the security group assigned to the Endpoint? – Stefano Zimmitti Mar 24 '22 at 14:15
  • Either Custom TCP, or just select HTTPS. You could also do Custom TCP and just select "ALL" for the ports. – Mark B Mar 24 '22 at 14:16
  • For the security group assigned to Endpoint, should I specify an Outbound rule? Or just leave it as it is? – Stefano Zimmitti Mar 24 '22 at 14:26
  • For now, all your security groups should just allow all outbound traffic. – Mark B Mar 24 '22 at 14:28
  • Just run the Lambda right now and still is going on time out – Stefano Zimmitti Mar 24 '22 at 14:43
  • 1
    I would probably spin up a tiny EC2 instance in the same VPC subnet, and use some tools like `dig` to verify the DNS is resolving to the endpoint, and try using the AWS CLI tool to send an SQS message from there. You could even test your Python code from there. Also, make sure the "Private DNS" setting is enabled for the VPC endpoint. – Mark B Mar 24 '22 at 15:00
  • I already had a running EC2 instance inside that VPC subnet. It works properly and SQS receive correctly the message, I used the following command: aws sqs send-message --region eu-west-1 --endpoint-url https://sqs.eu-west-1.amazonaws.com/ --queue-url https://sqs.eu-west-1.amazonaws.com/267684368050/tutorial-queue-test --message-body "Hello from Amazon SQS." – Stefano Zimmitti Mar 24 '22 at 16:09
  • Also EC2 has a security group that is not specified in the inbound rule's source of SQS's Endpoint. So I assume that there is something wrong with the lambda? – Stefano Zimmitti Mar 24 '22 at 16:18
  • You would need to create an EC2 instance without a public IP address to simulate the same conditions as the Lambda function. – Mark B Mar 24 '22 at 17:20
  • Make sure you have configured the VPC endpoint with the specific VPC subnets you are using, and make sure "private DNS" is enabled on the VPC endpoint. – Mark B Mar 24 '22 at 18:43
  • 1
    I'd expect to be able to work out what the root cause is without these tools, but be aware of [VPC Reachability Analyzer](https://docs.aws.amazon.com/vpc/latest/reachability/what-is-reachability-analyzer.html) and [Network Access Analyzer](https://docs.aws.amazon.com/vpc/latest/network-access-analyzer/what-is-vaa.html), both of which support VPC Endpoints. If you go that route, you might need to launch a small EC2 instance with the same SGs as the Lambda function. – jarmod Mar 24 '22 at 23:41
  • I've taken a bunch of screens about the current configurations, just to make sure that all is setup as you told me. [Current configurations](https://imgur.com/a/yXixlJk) – Stefano Zimmitti Mar 25 '22 at 09:34
  • If the solution isn't in any of the answers, add it as an answer yourself and accept it. – gshpychka Mar 25 '22 at 17:05

1 Answers1

0

Lambdas placed in a Public subnet do not have internet connectivity as they cannot use an Internet Gateway.

So the first step is to place the lambda in a Private or Isolated subnet, according to best practices.

Then, there are two options:

  1. If your Lambda requires internet connectivity, add a NAT gateway and add a route to it from the Private subnet. If you go this route, you will not need to use VPC endpoints.

  2. If you want to avoid using a NAT gateway, simply place the lambda in an Isolated subnet and it will use the VPC endpoints you created.

If the endpoints have private DNS enabled, you do not need to provide a custom endpoint_url, boto3 will work as is.

If this doesn't work, verify that the endpoint is created in the subnet that your Lambda is using. Also verify that you have Private DNS turned on for the endpoint.

See also: Why can't an AWS lambda function inside a public subnet in a VPC connect to the internet?

gshpychka
  • 8,523
  • 1
  • 11
  • 31
  • Can you cite any documentation that a VPC endpoint only works in private subnets? – Mark B Mar 24 '22 at 17:23
  • @MarkB I did not say it only works in private subnets. I said that by default, they're only enabled in Private subnets. You could enable them for Public subnets as well, but there's no reason to place a Lambda into a Public subnet. – gshpychka Mar 24 '22 at 17:28
  • The whole point of this question is to enable a Lambda to access SQS from a subnet that does not have a NAT gateway. In this scenario it shouldn't matter if the subnet has an Internet Gateway or not. I don't see why they should need to create more subnets in order to get the VPC Endpoint working. – Mark B Mar 24 '22 at 17:36
  • 1
    Some of the text here is misleading, I think. Interface VPC endpoints aren't "by default created in Private subnets" afaik. They're a VPC-level construct and, by default, no ENIs are created in any subnets when you create an Interface VPC Endpoint. Also, for each interface endpoint, you can choose only one subnet per AZ, so you could not choose both private & public subnets in the same AZ. – jarmod Mar 24 '22 at 18:35
  • @MarkB sure, it's just not a good practice to place Lambdas in public subnets, as it serves no purpose. If they don't have other subnets, I guess that would make sense. – gshpychka Mar 24 '22 at 18:38
  • @jarmod you're right, this is the default with CDK and I assumed the console follows the same behavior. I've updated the answer. – gshpychka Mar 24 '22 at 18:40