5

I have an application running in an EC2 instance in a private subnet (to add extra security), receiving traffic directly from an internet-facing NLB that is associated to the public subnet.

I have also configured a NAT Gateway in the public subnet so that the private EC2 instance can download whatever needed from the Internet.

I have just come out to the next conclusion:

  • If I request from the Internet: http://index.html

  • The private EC2 Instance uses the NAT Gateway to send back the HTTP response, and therefore you are charged for that NAT processing.

https://aws.amazon.com/vpc/pricing/

"Data processing charges apply for each Gigabyte processed through the NAT gateway regardless of the traffic’s source or destination"

The Route Table associated to the Private Subnet (where the web/app server is located) has { - local ; 0.0.0.0/0 - NATGateway} If I remove the 0.0.0.0/0 entry, then the HTTP requests to the server do not work. And if I remove the NAT Gateway I get the same issue. Additionally, when I have the NAT Gateway and the route table to use it from the Private Subnet, I can also see traffic in the monitoring tab of the NAT Gateway when I do a simple HTTP request from the Internet - http:///index.html

Has someone faced the same issue? Is my understanding correct?

Is there any workaround to avoid this? I can just think of the following:

  • Move the application to a Web Tier (in a public subnet) so that the EC2 Instance has a public IP and therefore it does not need the NAT Gateway to respond to every HTTP request

  • Create a Web Tier in addition to the Application Tier, so that all traffic goes from the NLB <-> Web Tier <-> App Tier

  • Create a NAT Instance instead of a NAT Gateway so that you are not charged for that NAT processing.

Thanks!

Luis
  • 595
  • 2
  • 7
  • 16

1 Answers1

10

So, it turns out there is an strange behaviour (bug?) in NLB that requires a default route to something for return traffic to work.

Scenario:

  • NLB in Public Subnet
  • NLB Target Group pointing to an instance in a Private Subnet

Sending traffic to the NLB will not give a response.

Add:

  • NAT Gateway
  • Add a route in the Private Route Table to point to the NAT Gateway

This works.

However, it doesn't actually use the NAT Gateway!

Instead of adding a NAT Gateway, you can instead create a default route to another instance, for example:

0.0.0.0/0 -> another instance

Then, connecting to the NLB works!

Behind the scenes, the VPC network detects that traffic coming out of the instance is in response to a request that came via the NLB, and traffic will be routed out the NLB.

HOWEVER, it only works if there is a valid default route in the route table. The route isn't used -- the network overrides it, so it doesn't matter what it points to. However, it must point to a valid resource. It doesn't work if it points to an ENI that isn't attached to anything. In my tests, it also doesn't seem to work if you point it back to the same instance. But, I pointed it to another instance and it worked just fine.

AWS Support agrees that this is not the best behaviour, but it's how things have to be configured for the moment.

Bottom line: It does not require a NAT Gateway. It just needs a valid default route that will be ignored when routing return traffic.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • Neat anomaly... but wait... OP reports that the traffic seems to appear in the NAT Gateway traffic stats. – Michael - sqlbot Feb 15 '18 at 00:44
  • Thanks John. But in your example, you are replacing the NAT Gateway for an EC2 Instance in the Public Subnet, and that instance becomes a point of failure. If you delete that EC2 Instance (or the NAT Gateway, or the 0.0.0.0/0 route) then the HTTP request from the Internet doesn´t work. Please when using a NAT Gateway, check the monitoring traffic when sending an HTTP request... it actually uses the NAT Gateway. – Luis Feb 15 '18 at 07:26
  • From what I was told, there merely has to be a valid default route. The NLB will send the return traffic. For example, I pointed the default route to an instance in the private subnet (with no NAT Gateway) and it worked. It didn't actually use the instance. – John Rotenstein Feb 16 '18 at 22:29
  • I wonder if it still works if you stop that EC2 instance that you are using for the default route. And also if that EC2 instance is actually receiving some traffic (wondering this because the NAT Gateway is actually receiving that traffic according to the Monitoring window) – Luis Feb 18 '18 at 15:44
  • I also had this confirmed by AWS support: "Since the instances in the target group won't send the response through the NAT Gateway and instead will go back through the NLB you won't be charged." – Alex Rudd May 22 '18 at 09:09
  • Which another instance? From same subnet? which ip you used.? – Metalhead Jan 24 '19 at 17:22