How to get a custom healthcheck path in a GCE L7 balancer serving a Kubernetes Ingress?

Question

I'm trying to deploy a grafana instance inside Kubernetes (server 1.6.4) in GCE. I'm using the following manifests:

Deployment (full version):

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: grafana
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: grafana
    spec:
      initContainers:
        …                                
      containers:
        - name: grafana
          image: grafana/grafana
          readinessProbe:
            httpGet:
              path: /login
              port: 3000
          …

Service:

apiVersion: v1
kind: Service
metadata:
  name: grafana
spec:
  selector:
    name: grafana
  ports:
    - protocol: TCP
      port: 3000
  type: NodePort

Ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: grafana
spec:
  tls:
    - secretName: grafana.example.com
  backend:
    serviceName: grafana
    servicePort: 3000

It turns out that grafana serves a 302 on / but the default GCE ingress healthcheck expects a 200 on / (source). As you can see, there is a custom readinessProbe in the Deployment (line 22).

Once I post these resources to the kube-apiserver, everything is created without error. Concretely, the Ingress gets a public ip4 address but the healtcheck is set up with the default / path instead of the custom one configured in the readinessProbe. Because of this, I get a 502 if I curl the public ip4 address of the Ingress.

The problem is fixable by manually changing the probe path to /login in the GCE console.

+1 same question. better than the login page, there's actually an `/api/health` [endpoint](http://docs.grafana.org/http_api/other/#health-api). — JCotton, Mar 15 '19 at 19:14

mmoya · Accepted Answer · 2020-04-16T11:06:59.063

16

Quoting from here:

The GLBC requires that you define the port (in your case 3000) within the Pod specification.

The solution is to declare the port used for the healthcheck in ports besides adding a custom readinessProbe:

containers:
  - name: grafana
    readinessProbe:
      httpGet:
        path: /login
        port: 3000
    ports:
      - name: grafana
        containerPort: 3000
    …

edited Apr 16 '20 at 11:06

answered Jun 20 '17 at 18:54

mmoya

1,901
1
21
30

It's worth pointing out that once the "backend service's health check parameters are inferred from a serving Pod's readiness probe, GKE does not keep the readiness probe and health check synchronized:". So updating the readinessProbe will not automatically correct the healthcheck. – ggordan Nov 08 '22 at 16:16

Tuxdude · Answer 2 · 2017-06-20T23:55:45.763

Customizing Health Checks

With GLBC addon

It is not quite clear from your question, but if you're using the GCE Load-Balancer Controller (GLBC) Cluster Addon, you can customize the health check path.

Currently, all service backends must satisfy either of the following requirements to pass the HTTP(S) health checks sent to it from the GCE loadbalancer:

Respond with a 200 on '/'. The content does not matter.

Expose an arbitrary url as a readiness probe on the pods backing the Service.

The Ingress controller looks for a compatible readiness probe first, if it finds one, it adopts it as the GCE loadbalancer's HTTP(S) health check. If there's no readiness probe, or the readiness probe requires special HTTP headers, the Ingress controller points the GCE loadbalancer's HTTP health check at '/'. This is an example of an Ingress that adopts the readiness probe from the endpoints as its health check.

The GLBC addon page mentions this in the Limitations section:

All Kubernetes services must serve a 200 page on '/', or whatever custom value you've specified through GLBC's --health-check-path argument.

Without GLBC addon

If you're not using the addon, currently Kubernetes does require you to serve 200 for GET requests on / path for successful health checks otherwise the backend will not get any traffic.

There is a bit of background about this in this bug.

Google Container Engine (GKE)

If you're on Google Container Engine (GKE), the same default Kubernetes requirements for health checks apply there too.

Services exposed through an Ingress must serve a response with HTTP 200 status to the GET requests on / path. This is used for health checking. If your application does not serve HTTP 200 on /, the backend will be marked unhealthy and will not get traffic.

Answer to your real issue

Having said all of this, as you (@mmoya) point out in your answer, adding the same port that is used for readiness probe as one of the ports in the pod fixes the issue for your case since the port itself is not exposed outside the pod otherwise. This caused Kubernetes to rely on the health check from / instead.

This answer basically says _You can't_ but it's actually possible. See my own answer. — mmoya, Jun 20 '17 at 19:02
@mmoya - Thanks for the link. I have now also added link to the doc from the GLBC repo where it describes why it could use `/` for certain cases and when readiness probes can be used for health checks. Since you did not mention GLBC in your question, I initially gave a generic answer which covered multiple cases. Your issue seems to be that you did not expose your port outside of the pod which is why it relied on the path `/` for health checks rather than use the readiness probes. — Tuxdude, Jun 20 '17 at 23:52

score 3 · Answer 3 · answered Jul 04 '21 at 13:02

the conditions for inferred healthchecks on Ingress creation are

an Ingress backend.servicePort references a Service port, matching a Pods spec.containers[].readinessProbe.httpGet.port and the Service's targetPort references the serving Pod's containers[].spec.ports.containerPort.

In mid 2020 GKE introduced an annotation and Custom Resource Definition BackendConfig to configure healthchecks explicitly, see concepts/ingress#health_checks.

Caveat: if you change the readinessProbe again though, on inferred healthchecks GKE does not keep the readiness probe and health check synchronized. It will only be inferred again on Ingress (re)creation.

To edit an external Loadbalancers healthchecks directly (for a custom http path), use

gcloud compute backend-services list
gcloud compute backend-services get-health BACKEND_SERVICE_NAME --global
gcloud compute health-checks describe
gcloud compute health-checks update http BACKEND_SERVICE_NAME --request-path=/api/health

this saved my day, for describe command also add the BACKEND_SERVICE_NAME — Nithya, Sep 28 '22 at 10:12

score 0 · Answer 4 · answered Aug 13 '21 at 07:26

This works on 1.9. Setting the httpHeaders also removes the necessity to add additional host name in ALLOWED_HOSTS setting.

readinessProbe:
  httpGet:
    path: /login
    port: 3000 # Must be same as containerPort
    httpHeaders:
    - name: Host
      value: yourdomain.com