7

Route table (in private subnet) won't change by adding VPCE as destination for aws secrets manager. Tried with new SG too (not using default SG). Any idea ?

rahul
  • 3,018
  • 4
  • 29
  • 28
  • What do you mean? You don't need to modify any RTs to use the endpoint. – Marcin Jan 23 '21 at 06:29
  • it suppose to show up as I create the endpoint right ? Like for S3 Gateway I see a new entry like: pl-63a5400a (com.amazonaws.us-east-1.s3, in destination and target = vpce id – rahul Jan 23 '21 at 06:32
  • 1
    S3 and DynamoDB are gateway VPC endpoints. These are first generation of endpoints that do modify RTs. SM uses interface endpoint (new generation) which does not do this. Does the SM endpoint work, or you can't connect to SM using it? – Marcin Jan 23 '21 at 06:37
  • 1
    Thanks a lot - you are right - It was not working because of some network issues (subnet mis config). But now it works and the route table changes won't show up. Learning = Gateways actually show a destination range in RT but not interfaces ! – rahul Jan 23 '21 at 06:44
  • 2
    Glad to hear it works. If you don't mind I will provide an answer with more details. – Marcin Jan 23 '21 at 06:59
  • How did it go? Is the issue still unresolved? – Marcin Apr 15 '21 at 10:47

2 Answers2

8

Based on the comments.

Secrets Manager (SM) uses VPC interface endpoints. This is new generation of endpoints, as compared to VPC gateway endpoints for S3 and DynamoDB. The new generation does not modify route tables (RTs). In contrast, the gateway endpoints do modify RTs specified when creating these endpoints.

For seamless work with the interface endpoints, it is important that the VPC has enableDnsHostnames and enableDnsSupport enabled, as well as private DNS for the endpoint. In addition, security group of the endpoint usually needs to be adjusted to allow connections on port 443.

Marcin
  • 215,873
  • 14
  • 235
  • 294
  • 3
    I can confirm that 443 inbound in the SG is needed for Secrets Manager to work. But do I really need all Source IPs? That seems a bit excessive of a rule just for Secrets Manager to work... Can I restrict them maybe? Thanks! – Peter Apr 30 '21 at 11:28
  • 1
    @Peter You can limit it to input from some SG or IP address. But endpoints only work for private IP address, so its not like you can access them from the internet. – Marcin Apr 30 '21 at 11:33
  • Yes thank you! I limited to SG itself and it still seems to work. TYVM – Peter Apr 30 '21 at 11:43
3

Recently I removed all NAT Gateways which originally allowed my private subnets to connect to the internet (including the Secrets Manager). I assumed that the NAT Gateway was a security vulnerability, expensive infrastructure and not required for private instances to contact the AWS Secrets Manager. I was wrong. My ECS task failed, and stated that the Secrets Manager resource could not be accessed.

Debugging

I recommend using the AWS VPC Reachability Analyzer to debug these network issues, it helped me.

  1. Launch an EC2 instance in the private subnet
  2. Assign your private subnet security groups to the EC2 instance
  3. Create a VPC Endpoint for your private subnets
  4. Use the VPC Reachability Analyzer to "Create and analyze path"
    • Source Type: Instance
      • Select the new EC2 instance you launched
    • Destination Type: VPC Endpoints
      • Select the VPC Endpoint you created
    • Destination port
      • Set to 443 for HTTPS
    • Protocol
      • Set to TCP
  5. Make changes to your infrastructure, and re-run the analyzer to test access until successful.

This will enable you to verify if the EC2 instance you launched in the private subnet can contact the VPC Endpoint (eg. Secrets Manager).

Checklist

Now that you have a means to quickly verify reach-ability, here are some steps that you might need to take in order to ensure your private subnet can contact AWS services like the Secrets Manager. Please consult AWS documentation as I may have outdated information.

Edward Corrigall
  • 414
  • 3
  • 8
  • I have decided not to use VPC Endpoints. In my case, to support Fargate, AWS requires me to have a VPC Endpoint for each of the supporting services: api.ecr, dkr.ecr, s3, etc. The cost for these endpoints is greater than the cost of a NAT gateway (or instance). Sounds pretty silly to pay so much to AWS to have access to AWS within AWS infrastructure. – Edward Corrigall Jul 13 '21 at 16:39
  • Thanks a bunch, very concise and easy to follow. – Greg Biles Sep 20 '22 at 00:29
  • Super, duper helpful thank you dude! – ronald mcdolittle Feb 20 '23 at 22:25