49

I am trying to set up Elastic Load Balancing (ELB) in AWS to split the requests between multiple instances. I have created several images of my webserver based on the same AMI, and I am able to ssh into each individually and access the site via each distinct public DNS.

I have added each of my instances to the load balancer, but they all come back with the Status: Out of Service because they failed the health check. I'm mostly confused because I can access each instance from its public DNS, but I get a timeout whenever I visit the load balancer DNS name.

I've been trying to read through all the docs and googling it, but I'm stuck. Any pointers or links in the right direction would be greatly appreciated.

Steffen Opel
  • 63,899
  • 11
  • 192
  • 211
jkeesh
  • 3,289
  • 3
  • 29
  • 42

14 Answers14

24

I contacted AWS support about this same issue. Apparently their system doesn't know how to handle cases were all of the instances behind the ELB are stopped for an extended amount of time. AWS support can manually refresh the statuses, if you need them up immediately.

The suggested fix it to de-register the ec2 instances from the ELB instead of just stopping them and re-register them when you start again.

batman 74
  • 396
  • 2
  • 4
  • 5
    I have the same problem, removing the instances (that were stopped when I created the ELB) and adding when they are running, doesn't seem to solve the issue. I still get _Out of Service_ – Parmaia Dec 02 '13 at 10:38
  • This is the same solution I found that worked for me. – retromuz Mar 18 '14 at 08:04
  • It sucks that we have to have manual interaction in order to get the ELB back into a default working state if one of the instances goes down. It seems to me it should be able to notify the ELB that it's back online and the ELB marks it back in service. I shudder to think what would happen in the off chance that if you only have 2 instances (varying AZ) and they both went down overnight... – Mike Purcell Nov 05 '15 at 03:37
20

Health check is (by default) made by accessing index.html on each instance incorporated in load balancer. If you don't have index.html in document root of instance - default health check will fail. You can set custom protocol, port and path for health check when creating elastic load balancer.

ivankoni
  • 511
  • 1
  • 3
  • 12
  • Also if you haven't yet installed apache/nginx.. Cheers for tip. – geotheory Jun 28 '18 at 14:13
  • this was my issue, my entire web application required a login, so health check was getting a 302 redirect to the login page! – Chad Dec 16 '19 at 14:59
11

Finally I got this working. The issue was with the Amazon Security Groups, because I've restricted the access to port 80 to few machines on my development area and the load balancer could not access the apache server on the instance. Once the load balancer gained access to my instance, it gets In Service.

I checked it with tail -f /var/log/apache2/access.log in my instance, to verify if the load balancer was trying to access my server, and to see the answer the server is giving to the load balancer.

Hope this helps.

Parmaia
  • 1,172
  • 1
  • 13
  • 28
  • I got this "GET / HTTP/1.1" 302 625 "-" "ELB-HealthChecker/1.0"... can you help me in that – Upendra Sep 04 '14 at 12:35
  • That means that the LoadBalancer got access to the instance. It should be **In Service** on any moment. – Parmaia Sep 05 '14 at 17:38
  • It might help someone else. Make sure to also check your elb security group, mine didn't allow outbound traffic from the elb to ec2 instances. – Jonathon Rossi Mar 17 '15 at 07:19
  • 2
    On the EC2 console I set `Load Balancers -> Health Check -> Edit Health Check -> Ping Path: /index.html`. I read the error logs in `sudo tail -f /var/log/httpd/error_log` and needed to `touch /var/www/html/index.html` because the error said `Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html,index.php) found, and server-generated directory index forbidden by Options directive`. 30 seconds later my load balancers were healthy. – hexicle Mar 09 '17 at 14:25
  • I had this issue, and it was due to both my inbound and outbound rule only allowing HTTP on port 80. I needed to add another rule for HTTPS on port 443. – C Murphy Aug 04 '20 at 11:22
4

If your web server is running fine, then it means the health check goes on a url that doesn't return 200.

A trick that works for me : go on the instance, type curl localhost:80/pathofyourhealthcheckurl

After you can adapt your health check url to always have a 200 response.

user1384132
  • 51
  • 7
  • 1
    That really helped me find my issue: I had a .htaccess password protection enabled on my ec2 instance. Of course the loadbalancer couldn't get passed that. So it got 401 back. :-) – konse Dec 07 '17 at 18:54
1

In my case, the rules on security groups assigned to the instance and the load balancer were not allowing traffic to pass between the two. This caused the health check to fail.

bitstream
  • 459
  • 9
  • 20
1

I to faced same issue , i changed Ping Protocol from https to ssl .. it worked !

Go to Health Check  --> click on Edit Health Check -- > change Ping protocol from HTTPS to SSL

Ping Target SSL:443
Timeout 5 seconds
Interval    30 seconds
Unhealthy Threshold 5
Healthy Threshold   10
Giri
  • 11
  • 1
1

For anyone else that sees this thread as this isn't listed:

Check that the health check is checking the port that the responding server is listening on.

E.g. node.js running on port 3000 -> Point healthcheck to port 3000;

Not port 80 or 443. Those are what your ALB will be using.

I spent a morning on this. Yes.

user1429671
  • 69
  • 6
0

I would like to provide you a general way to solve this problem. When you have set up you web server like apache or nginx, try to read the access log file to see what happened. In my occasion, it report 401 error because I have add the basic auth in nginx. Of course, just like @ivankoni remind, it may because of the document you check is not exist.

blio
  • 473
  • 5
  • 12
0

I was working on the AWS Tutorial on hosting a web app and ran into this problem. Step 7b states the following:

"Set Ping Path to /. This sends queries to your default page, whether it is named index.html or something else."

They could have put the forward slash in quotations like this "/". Make sure you have that in your health checks and not this "/." .

nalyd88
  • 4,940
  • 8
  • 35
  • 51
Scott123180
  • 91
  • 1
  • 11
0

Adding this because I've spent hours trying to figure it out...

If you configured your health check endpoint but it still says Out of Service, it might be because your server is redirecting the request (i.e. returning a 301 or 302 response).

For example, if your endpoint is supposed to be /app/health/ but you only enter /app/health (no trailing slash) into the health check endpoint field on your ELB, you will not get a 200 response, so the health check will fail.

Nick Yap
  • 887
  • 1
  • 10
  • 13
0

I had a similar issue. The problem appears to have been caused due to my using a HTTP health check and also using .htaccess to password protect the site.

HardlyNoticeable
  • 497
  • 9
  • 26
0

I got the same error, in my case had to copy the particular html file from s3 bucket to "/var/www/html" location. The same html referenced in load balancer path.

The issue resolved after copying html file.

SuperSA
  • 1
  • 1
0

I had this issue too, and it was due to both my inbound and outbound rule for the Load Balancer's Security Group only allowing HTTP traffic on port 80. I needed to add another rule for HTTPS traffic on port 443.

C Murphy
  • 313
  • 2
  • 11
0

I was also facing that same issue, where ELB (Classic-Load-Balancer) try to request /index.html not / (root) while health check. If it unable to find /index.html resource it says 'OutOfService'. Be Sure index.html should be available.

Saurabh Bhurle
  • 169
  • 10