75

I'm using Amazon EC2, and I want to put an internet-facing ELB (load balancer) to 2 instances on a private subnet. I am using VPC with public and private subnets.

  • If I just add the private subnet to the ELB, it will not get any connections.
  • If I attach both subnets to the ELB then it can access the instances, but it often will get time-outs. (Refer Screenshot 1)
  • If I attach to only public subnet then my instance attached to ELB gets OutOfService because I do not have any instance in the Public Subnet, instance count shows 0. (Refer Screenshot 2)

Screenshot 1: Both subnets attached Both subnets attached

Screenshot 2: Only public subnet attached Only public subnet attached

My question is actually an extension to this question. After following all 6 steps mentioned in the accepted answer, I am still getting struck, my instance attached to ELB gets OutOfService. I have even tried with allowing ports in the Security Groups for EC2 instances and ELB, but it did not help.

Please help, I am breaking my head with this.

Community
  • 1
  • 1
Manish Singh
  • 5,848
  • 4
  • 43
  • 31

2 Answers2

107

The other SO question you referenced is spot on. Double/Triple check the following

  • You need to attach only public subnets to your ELB, making sure that the availability zones those subnets are aligned with the availability zones of the private subnets that your instances are in.
  • Make sure that the security group of your instances allows access from the security group of your load balancer
  • The load balancer security group should have an egress rule allowing the health check to reach the instance
  • Make sure that your health check is working locally on the instance. For example, if your health check in the ELB is HTTP:8080/health_check, on the instance you can curl x.x.x.x:8080/health_check (where x.x.x.x is the private IP of the instance) and get a 200 response code.
  • The public subnet routing table should route 0.0.0.0/0 to the internet gateway attached to your VPC.
  • The private subnet routing table should route 0.0.0.0/0 to a NAT instance or gateway in a public subnet
Ben Whaley
  • 32,811
  • 7
  • 87
  • 85
  • Hi Ben, thanks for your answer. I have triple checked all the points you mentioned and all of them seem to be in place. I have attached some screenshots to the question, could you please check them. When I attach only public subnets to the ELB, then the instance count is 0 and the status is `OutOfService`. If I attach both subnets then the health check is working fine, but the ELB often gives timeouts on connecting. (Here in the screenshots, I am adding only one instance to the ELB). – Manish Singh Mar 20 '14 at 19:16
  • 2
    I notice in screenshot 1 that the healthy instance is in AZ ap-southeast-1b. Make sure that the public subnet for you ELB is in the same zone. E.g. public subnet for ELB is in ap-southeast-1b, and private subnet for instance is in ap-southeast-1b. If the zones do not match it will not work. – Ben Whaley Mar 20 '14 at 19:54
  • 3
    @BenWhaley is correct. To expand this just a bit: Your ELB should only be on public subnets, and only on subnets in the same availability zone(s) as your instance(s). – Michael - sqlbot Mar 20 '14 at 21:16
  • Thanks Ben and Michael, a million times! My public and private subnets were in different availability zones. That was the problem. – Manish Singh Mar 21 '14 at 03:33
  • 1
    So doesn't the "CrossZone" setting on the ELB allow the ELB and the instances to be in different AZs? It doesn't appear to actually work for me. Not a major issue anyway, as I'll just ensure AZ-alignment. – d11wtq Apr 09 '15 at 04:39
  • 2
    Cross Zone ensures that a load balancer interface within one zone will send traffic to instances in other AZs. [Docs](http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-disable-crosszone-lb.html) – Ben Whaley Apr 09 '15 at 04:47
  • Very helpful. Thank's for this – jmcollin92 Oct 07 '15 at 20:29
  • 2
    The reason for this requirement is that AWS puts an ELB "node" in each subnet associated with the ELB. The network interfaces of these nodes have public IPs and need to be able to talk to the Internet, and thus need to be in a subnet with an IGW route. – seanmcl May 23 '16 at 19:52
  • If traffic is coming from the ELB to the EC2 instances, does the reply from the EC2 instances go back to the ELB? If it does, this means that the ELB would then forward to the response back to the client. If this is true, then instead of needing a NAT instance or gateway, couldn't we just allow inbound HTTP/HTTPS access on our private EC2 instances security group from the security group that the ELB sits in and the rest would be taken care of? – havak5 Mar 29 '17 at 20:52
  • 1
    Yes, if you don't need other outbound access from your private instances then you don't need a NAT. – Ben Whaley Mar 29 '17 at 23:52
  • 1
    I've found this official source pretty dead-on, but this question helped me prove to my DevOps team that their setup was causing us problems in our QA environment, so thanks! https://aws.amazon.com/premiumsupport/knowledge-center/public-load-balancer-private-ec2/ – Brett Green Oct 16 '17 at 14:27
  • What do you mean by zones should "align"? – A. Kutluozen Nov 29 '17 at 19:58
  • I mean that if you have instances in zones A and B, make sure your load balancer is present in subnets in zones A and B. If it is instead in only zone C, things will not work. – Ben Whaley Nov 30 '17 at 20:27
  • 1
    One extra thing to check: the load balancer security group should have an egress rule allowing the health check to reach the instance – dancl Jan 06 '20 at 14:46
  • @BenWhaley Will the load balancer will not work if Load balancer is attached with private subnets? – Raphael Titus Aug 20 '20 at 15:09
  • This link might be helpful, "Public load balancer to private subnet ec2" https://aws.amazon.com/premiumsupport/knowledge-center/public-load-balancer-private-ec2/ – smile Oct 28 '20 at 21:23
4

The other SO question helped me as well. For me I kept forgetting that I needed to install software to server (i.e Apache) and if you don't create a NAT GW or some other method to allow software installs it will fail. If you want to try an automated fashion w/o much hassle, you can try a script:

https://github.com/jouellnyc/AWS/tree/master/create_aws_vpc3

Note there's plenty of good docs and discussions already but:

HTTP codes I got from the ELB:

  • 504 - Probably security groups - allow access to the port 80 of the instances
  • 503 - Probably the wrong target group setup
  • 502 - Probably Apache/Server not running b/c it’s not installed because there's no nat gw or method to install software.
jouell
  • 3,288
  • 1
  • 18
  • 15
  • I had an EC2 in private subnet without NAT, and an ELB in public subnet with IG. Couldn't figure out for the life of me why I kept getting 502 errors. I was deploying through CDK and your answer made me realize I need to attach a NAT at least initially so that necessary software gets installed on instance. Thanks a bunch! – Akash Jul 27 '22 at 07:38