1

I am having two vpcs, one is a mgmt vpc with all the necessary vpc interface endpoints connecting to AWS services, and another peered vpc established peering connection with the mgmt vpc. I have tried to ssh into my instance (call it instance A) in the peered vpc and call aws api through cli with the following command

aws ec2 describe-instances

but still the connection just timeout everytime.

I have checked my vpc endpoint security group and double confirmed I have set it to allow all incoming traffic from the security group attached to instance A. And my instance A's security group has set to allow all traffic egress to the vpce security group.

Anyone knows or encounter this issue before? Anything that I missed out or did wrong?

EDIT: My peered vpc has several subnets, with the only IGW located in the ingress/egress tier subnets. In the web tier subnets, there are the instances that I am trying to call the vpce there, and the subnets here have NAT gateway.

For mgmt vpc the only way to ssh in is through a jumphost instance inside the vpc.

The routing tables for web tier subnets are as follows:

Destination            Target
100.113.189.0/24       pcx-0d3974s489064s3sd
100.113.206.0/24       local
10.196.162.128/25      local

Security group for web tier subnet instances are as follows:

 Outbound
 Port Range             Protocol      Source
 All                    All           sgrp-<vpce_to_ec2>

The routing table for the subnet with vpce in mgmt vpc:

Destination            Target
100.113.206.0/24       pcx-0d3974c6890640bd2
100.113.189.0/24      local
10.196.157.128/25     local
pl-6fa54006           vpce-<this_is_for_s3>

For the vpce to ec2 security group:

Inbound
Port Range             Protocol      Source
All                    All           sgrp-<web_tier_instance>
All                    All           100.113.189.0/24

Note that each vpc has assigned with two cidr blocks. Also there is s3 endpoint gateway in mgmt vpc as well

tnkh
  • 1,749
  • 2
  • 14
  • 30
  • 2
    Have you tried by specifying the VPC endpoint DNS name: `aws ec2 describe-instances --endpoint-url vpce-05c21657a045fff54-puytslup.ec2.us-east-1.vpce.amazonaws.com`. Obviously, your endpoint DNS name will be different. You have to check it in the vpc interface endpoint details. – Marcin Feb 20 '21 at 14:37
  • @Marcin Yea I just tried with your suggestion but it still hangs then timeout. My routing are ok with symmetrical connection on both vpcs connecting to each other through peering id. Do I need to add extra route to set destination as aws api and target as peering id too in my peered vpc? – tnkh Feb 21 '21 at 00:19
  • 1
    Can you update the question with VPC settings. Are these VPCs fully private, no internet gateway? What are your route tables and everything else. I can try to re-construct your VPC peering architecture and reproduce the issue. – Marcin Feb 21 '21 at 02:39
  • @Marcin I have updated with the details. Please let me know if you need any more details. Many thanks ! – tnkh Feb 21 '21 at 03:16
  • 1
    I created the two VPCs to have one private VPC only with EC2 endpoint, and webteir vpc. The use of `--endpoint-url` works as expected. Without `--endpoint-url` the aws cli hangs. Thus you should double check your route tables and security groups. – Marcin Feb 21 '21 at 03:54
  • @Marcin Many thanks for what you have helped. Weird, I recreated and tested again but still the aws cli hangs. I wonder how u did differently than me. You used security group id to indicate the source/destination when configuring security group right? – tnkh Feb 21 '21 at 11:54
  • 1
    Have you enabled, in both VPCs, DNS hostnames and DNS support? Also can you verity that your VPCE works at all, for example, by creating instance in mgmt and trying aws cli there? – Marcin Feb 21 '21 at 12:04
  • @Marcin yes I have enabled both VPCs DNS support and hostnames, my mgmt vpc can call the ec2 api with aws cli too. – tnkh Feb 21 '21 at 13:04
  • 1
    @Marcin I tried again today and tested with what you suggested and things work. Now I am able to call the vpce endpoint in my peered vpc by specifying the endpoint url explicitly. Many thanks. Anyway I can credit you besides upvoting your comment? – tnkh Feb 25 '21 at 07:12

1 Answers1

1

Based on the comments.

I tried to reproduce the issue and the OP's architecture and can verity that the connection to endpoint works if --endpoint-url is used.

aws ec2 describe-instances --endpoint-url vpce-05c21657a045fff54-puytslup.ec2.us-east-1.vpce.amazonaws.com

In the above, the endpoint url (vpce-05c21657a045fff54-puytslup.ec2.us-east-1.vpce.amazonaws.com) can be obtained from VPC Interface details.

It maybe possible that enabling DNS Resolution Support for peering connection also works. In that case --endpoint-url may be not needed. However, I haven't verified that in my tests, as I only focused on solving the issue with --endpoint-url.

Marcin
  • 215,873
  • 14
  • 235
  • 294
  • Hi sorry for late response. Btw I have tested the DNS resolution support you suggested here. Both the DNS resolution and support are enabled for both vpcs, but I still need to specially specify --endpoint-url to call aws endpoint. – tnkh Mar 07 '21 at 08:33
  • @tnkh Thanks for letting me know. So I guess this is the using `--endpoint-url ` is the only way. – Marcin Mar 07 '21 at 08:43