Custom Health check with GCP

Question

Hi I try to use custom health check with GCP LoadBalancer.

I have added readinessProbe & livenessProbe like this:

    readinessProbe:
      httpGet:
        path: /health
        port: dash
      initialDelaySeconds: 5
      periodSeconds: 1
      timeoutSeconds: 1
      successThreshold: 1
      failureThreshold: 10
    livenessProbe:
      httpGet:
        path: /health
        port: dash
      initialDelaySeconds: 5
      periodSeconds: 1
      timeoutSeconds: 1
      successThreshold: 1
      failureThreshold: 10

But when I create my ingress I haven't got my custom health check

Path LB

Back in the days this was possible. Not sure if it is still supported. If it is, what you have to do is to set up a liveness probe. If the liveness probe goes through, then GCP Load Balancer would adopt that endpoint for the health check of the nodes. Is your liveness probe going through? Do you get 200? — suren, Apr 08 '19 at 10:39
I had a similar problem and noticed the readinessProbe needs to by defined before creating Ingress resource - it won't pick up later changes. Is it possible this is your case as well? — Robert Lacok, Apr 08 '19 at 10:55
I tried to wait 10 minutes before creating the ingress but no effect .. — M.Hol, Apr 08 '19 at 12:37
https://github.com/kubernetes/ingress-gce/blob/0d15eedbefb931b401ef9fc1d6e1a7d73c08dbc4/pkg/backends/backends.go#L570 "Kubernetes L7 health check generated with readiness probe settings."... — M.Hol, Apr 08 '19 at 12:42

M.Hol · Accepted Answer · 2019-04-09T09:44:50.350

I FINALIZED an answer. What I was trying to do was impossible. My GCE Ingress used a backend on port 80 . But in my ReadinessProbe I told him to check on port 8080 and on the /health path. This is impossible!

The port of the service declared in the Ingress backend must be the same as that declared in the readinessProbe. Only the path can be different. If we do not respect this pattern, it is / that is associated with the Health Check GCP path.

From a network point of view this is logical, the Health Check GCP is "out" of the Kube cluster, if we tell it to route on port 80 but our ReadinessProbe is on another port, how it can ensure that even if the port associated with the ReadinessProbe meets port 80 (which is the one on which it must route traffic) also respond.

In summary, the port of the backend declared in Ingress must have a readinessProbe on the same port. The only thing we can customize is the path.

score 1 · Answer 2 · answered Apr 08 '19 at 09:42

1

I think you are confused between resources in GCP.

The code you posted is at no moment in relation to a Load balancer resource, as it's a kubernetes health check for pod states. If you want to know if the probes are working, check your pod state, if it's not running describe your pod and look at the logs, should indicate an issue with the probes.

I'm going to guess that you have an ingress resource somewhere in your kubernetes conf wich creates the lb and all the resources around it like the health check (still guessing that the image you posted is in relation to that).

If you are using GKE you should leave the google automated resource conf from k8s config you deployed as it is, cause you may brake some things that google is already maintaining for you.

answered Apr 08 '19 at 09:42

night-gold

2,202
2
20
31

When I create an Ingress on GKE with GCE class, LB create backend and health check. The health check is binding on `/`path. But let’s take an example, we have an app that on the path `/` makes a redirect ( `302` ) on/dashboard. Since the returned HTTP code is not `200` , our service returns `UNHEALTHY` . I know it is possible to modify the Endpoint of the healthcheck by hand but I would like to do it automatically and especially as the heathcheck corresponds to my Readiness . – M.Hol Apr 08 '19 at 09:52
I may have said something wrong: you can find the same question as yours here: https://stackoverflow.com/questions/50018193/gce-ingress-not-picking-up-health-check-from-readiness-probe?rq=1 but there is no validated answer... – night-gold Apr 08 '19 at 10:22
No problem. I just find a possible answer : https://github.com/pusher/oauth2_proxy/issues/86 & https://github.com/jetstack/kube-lego/issues/27 . But I think GCP Ingress is constraining... – M.Hol Apr 08 '19 at 10:36

Custom Health check with GCP

2 Answers2