Use of security group for AWS Secrets Manager endpoint

Question

I need to create an endpoint to access AWS Secrets Manager using CloudFormation for rotating secrets. At Using an AWS Secrets Manager VPC endpoint it says:

We recommend that you create a Secrets Manager endpoint in the same VPC so that requests from the Lambda rotation function to Secrets Manager don't leave the Amazon network.

At Access an AWS service using an interface VPC endpoint > Create a VPC endpoint it says:

For Security group, select the security groups to associate with the endpoint network interfaces. The security group rules must allow resources that will use the VPC endpoint to communicate with the AWS service to communicate with the endpoint network interface.

But it doesn't give me any insight into what specific rules I need to set up in this situation.

In the CloudFormation documentation to create an endpoint, AWS::EC2::VPCEndpoint SecurityGroupIds, it says:

The IDs of the security groups to associate with the endpoint network interfaces. If this parameter is not specified, we use the default security group for the VPC. …

The examples at AWS::SecretsManager::RotationSchedule in fact show creation of a secrets manager endpoint, explicitly indicating the default security group for the VPC. (If that is the default, I don't know why they specify it.)

  SecretsManagerVPCEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      SubnetIds:
      - Ref: TestSubnet01
      - Ref: TestSubnet02
      SecurityGroupIds:
      - Fn::GetAtt:
        - TestVPC
        - DefaultSecurityGroup
      VpcEndpointType: Interface
      ServiceName:
        Fn::Sub: com.amazonaws.${AWS::Region}.secretsmanager
      PrivateDnsEnabled: true
      VpcId:
        Ref: TestVPC

My question is if I want to rotate passwords using a lambda, what security group should I associate with the secrets manager endpoint? Is it most appropriate just to use the VPC default security group? Or is there some more specific security group that I should create to be more secure, and if so, what rules should it have?

Answer 1

In my opinion, while using the default security group for pretty much everything is certainly convenient, it is not a best security practice.

If you want to move away from the default security group to a more defense in depth architecture, then ideally you would allow only the traffic absolutely necessary between components of your architecture. In AWS that typically is done through firewalls (called Security Groups in AWS).

For your specific case where a given Lambda function needs private access to both the Secrets Manager service and to a private database endpoint:

1. Create a new Security Group, associate it with the VPC Endpoint and disassociate the default Security Group from the VPC Endpoint. Now you have a more specific Security Group.

2. Create another new Security Group, associate it with your Lambda function and disassociate the default Security Group from the Lambda function.

3. Add an ingress rule to the VPC Endpoint Security Group that allows traffic from the Lambda function's Security Group. Specifically allow ingress on port 443 from the Security Group associated with your Lambda function: TCP, port 443, and source is the ID of the Lambda function's Security Group. That will allow the Lambda function to make Secrets Manager API calls (which are made over HTTPS) to the Secrets Manager service via your VPC Endpoint.

4. Repeat step 3, but allowing your Lambda function Security Group to send traffic to your (new, if needed) database Security Group. Be sure to configure the relevant protocol and port(s) for your database.

5. Your Lambda function Security Group also needs to allow outbound traffic, of course. Typically you would start with allow all outbound traffic on all protocols, all ports. See the default Outbound rules documented here. You can tailor these as needed later, e.g. to restrict the Lambda function's outbound traffic to the VPC Endpoint and your database, as needed.

Here's a complete working example of a YAML CloudFormation template with an inline Lambda function configured to read a secret from Secrets Manager via a VPC Endpoint. Modify or provide an appropriate value for the SecretName parameter.

AWSTemplateFormatVersion: "2010-09-09"
Description: Stack Overflow 75966588

Parameters:
  SecretName:
    Description: The name of the secret to read
    Type: String
    Default: my-secret

Resources:
  # VPC
  Vpc:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16
      EnableDnsHostnames: True
      EnableDnsSupport: True
  # Private Subnet
  PrivateSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      CidrBlock: 10.0.1.0/24
      VpcId: !Ref Vpc
  # Route Table
  PrivateRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref Vpc
  # Associate Private Subnet with Private Route Table
  AssociateRouteTable:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PrivateRouteTable
      SubnetId: !Ref PrivateSubnet
  # Security Group for Lambda function
  LambdaSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security Group for Lambda Function
      VpcId: !Ref Vpc
  # Security Group for VPC Endpoint allowing ingress from Lambda SG
  VpcEndpointSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security Group for Secrets Manager Endpoint
      VpcId: !Ref Vpc
      SecurityGroupIngress:
        - IpProtocol: "-1"
          SourceSecurityGroupId: !GetAtt LambdaSecurityGroup.GroupId
  # VPC Endpoint to access Secrets Manager
  SecretsManagerEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      PrivateDnsEnabled: True
      SecurityGroupIds:
        - !GetAtt VpcEndpointSecurityGroup.GroupId
      ServiceName: !Sub com.amazonaws.${AWS::Region}.secretsmanager
      SubnetIds:
        - !Ref PrivateSubnet
      VpcEndpointType: Interface
      VpcId: !Ref Vpc
  # IAM role for Lambda function
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      Path: /
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
        - arn:aws:iam::aws:policy/SecretsManagerReadWrite
  # Inline Python Lambda function
  LambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: !Sub |
          import json
          import boto3

          def get_secret(secret_name):
              client = boto3.client('secretsmanager')
              return client.get_secret_value(SecretId=secret_name)['SecretString']

          def lambda_handler(event, context):
              print("Secret:", get_secret("${SecretName}"))
              return {
                  'statusCode': 200,
                  'body': json.dumps('Hello from Lambda!')
              }
      Handler: index.lambda_handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Runtime: python3.10
      Timeout: 10
      VpcConfig:
        SecurityGroupIds:
          - !Ref LambdaSecurityGroup
        SubnetIds:
          - !Ref PrivateSubnet