1

I have a k8s ingress with more than 5 backend services behind it. The ingress spawns a GoogleCloud LoadBalancer.

Each of the services is routed traffic to by an http path rule. Eg. one app is on /foo, another is on /bar, etc. All of them work fine. Then I added a new app, with backend service and routing rule, all the same way as the others.

But I'm constantly getting this error when I hit the URL of the new app:

Error: Server Error
The server encountered a temporary error and could not complete your request.
Please try again in 30 seconds.

When I open the ingress in GCP console, I can see this warning: enter image description here

and the unhealthy service is the one from my newly added app.

The weird thing is that the app actually does get traffic when I hit the URL. I can see it in the logs. But I still get that 502 error and the backend service is shown as unhealthy.

I am not rly sure how to debug this in order to figure what's the issue.

Milkncookiez
  • 6,817
  • 10
  • 57
  • 96
  • 1
    does the health check url of your backend service return a 200 response? It may be possible that the health check is failing because your app is redirecting to the login page(302 response) on your liveness/readiness probe. – dishant makwana Jun 11 '21 at 10:40
  • I don't have a health-check endpoint on this app. It's just a node.js server that servers static files. It's configured that a hit on any endpoint (aka `*`) will either redirect to a different app (if no access token available) or just serve the static files. – Milkncookiez Jun 11 '21 at 15:28
  • 1
    When you set up an ingress resource, GCP creates health checks for all backend services which are part of the loadbalancer. The default path is '/' root url. If you want the health check url to be something else, you need to attach a readiness probe to your pods. – dishant makwana Jun 11 '21 at 15:44
  • 1
    As others commented, if the backend receives traffic AND the load balancer reports UNHEALTHY, your health check is failing. Once you solve that problem, post an answer showing the problem and solution for the benefit of others. – John Hanley Jun 11 '21 at 19:18
  • I can see [in the GCP ingress docs](https://cloud.google.com/kubernetes-engine/docs/concepts/ingress#health_checks) that the default path is `/` root URL for the healthcheck. But if I've got `readiness` probe, the ingress will redirect its healthcheck to the path of the `readiness` probe. I added the probe, but the service is still stuck in `UNHEALTHY` state. And when I open the healthcheck object in GCP console, it still has `/` as its path. – Milkncookiez Jun 13 '21 at 16:01
  • I restarted the ingress, hoping it will update the healthcheck object, picking up the availability of the `/readiness` probe and endpoint. But it didn't. So, I had to manually update the `path` to which the healthcheck object of the backend service was hitting. I guess the pod with `readiness` probe should exist before the ingress is set up, otherwise it doesn't update the healthcheck object automatically. – Milkncookiez Jun 13 '21 at 17:11
  • There is a [same error's post](https://stackoverflow.com/questions/62744880/error-server-error-the-server-encountered-a-temporary-error-and-could-not-compl). But I am not sure if your case issue is related to the [documentation's limits](https://cloud.google.com/kubernetes-engine/docs/concepts/ingress#limitations) "Combining multiple Ingress resources into a single Google Cloud load balancer is not supported." – Pit Jun 14 '21 at 08:09

1 Answers1

5

So, the problem was that the LB health-check was hitting / which was a non-existing endpoint on the app (aka. it wasn't returning OK 200).

I added readiness probe to the k8s Deployment. According to the GCP Ingress docs, if there is a readiness probe, the ingress will pick it up and use it as the LB health-check.

I also had to manually update the path to which the health-check object of the backend service was hitting. I guess the pod with readiness probe should exist before the ingress is set up, otherwise it doesn't update the health-check object automatically.

Milkncookiez
  • 6,817
  • 10
  • 57
  • 96