GCE Ingress not picking up health check from readiness probe

Question

When I create a GCE ingress, Google Load Balancer does not set the health check from the readiness probe. According to the docs (Ingress GCE health checks) it should pick it up.

Expose an arbitrary URL as a readiness probe on the pods backing the Service.

Any ideas why?

Deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: frontend-prod
  labels:
    app: frontend-prod
spec:
  selector:
    matchLabels:
      app: frontend-prod
  replicas: 3
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: frontend-prod
    spec:
      imagePullSecrets:
        - name: regcred
      containers:
      - image: app:latest
        readinessProbe:
          httpGet:
            path: /healthcheck
            port: 3000
          initialDelaySeconds: 15
          periodSeconds: 5
        name: frontend-prod-app
      - env:
        - name: PASSWORD_PROTECT
          value: "1"
        image: nginx:latest
        readinessProbe:
          httpGet:
            path: /health
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5
        name: frontend-prod-nginx

Service:

apiVersion: v1
kind: Service
metadata:
  name: frontend-prod
  labels:
    app: frontend-prod
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: frontend-prod

Ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: frontend-prod-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: frontend-prod-ip
spec:
  tls:
    - secretName: testsecret
  backend:
    serviceName: frontend-prod
    servicePort: 80

Do you have anything serving on http://:3000/healthcheck and http://:80/health? — suren, Apr 26 '18 at 10:15
There are some limitations here[1]. I am going to reproduce your use case. [1]: https://github.com/kubernetes/ingress-gce/blob/master/examples/health-checks/README.md — suren, Apr 27 '18 at 13:10
Awesome. I'm using GKE with latest version of kubernetes they provide no alpha features. — Amarjeet Singh Rai, Apr 27 '18 at 15:06
Hi there. So, I did the test, and it worked. Now, when describing the ingress, there is nothing there. I went to the LB page > health checks, in GCP and there is was; a note saying "Kubernetes L7 health check generated with readiness probe settings." I made the test with one nginx container though. May be it is necessary to run a test with two containers; one of them on the service port, the other one on random port. — suren, May 01 '18 at 14:55

Amarjeet Singh Rai · Answer 1 · 2022-06-12T14:40:18.937

15

So apparently, you need to include the container port on the PodSpec. Does not seem to be documented anywhere.

e.g.

    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

Thanks, Brian! https://github.com/kubernetes/ingress-gce/issues/241

edited Jun 12 '22 at 14:40

answered May 02 '18 at 15:55

Amarjeet Singh Rai

881
2
9
20

still true in 2022, but at least it's documented in a barely-visible note here: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress#health_checks – Andreas Jansson Aug 04 '22 at 22:03

ahmet alp balkan · Answer 2 · 2020-04-15T22:22:54.253

5

Update by Jonathan Lin below: This has been fixed very recently. Define a readinessProbe on the Deployment. Recreate your Ingress. It will pick up the health check path from the readinessProbe.

GKE Ingress health check path is currently not configurable. You can go to http://console.cloud.google.com (UI) and visit Load Balancers list to see the health check it uses.

Currently the health check for an Ingress is GET / on each backend: specified on the Ingress. So all your apps behind a GKE Ingress must return HTTP 200 OK to GET / requests.

That said, the health checks you specified on your Pods are still being used ––by the kubelet to make sure your Pod is actually functioning and healthy.

edited Apr 15 '20 at 22:22

answered May 01 '18 at 20:21

ahmet alp balkan

42,679
38
138
214

1

Has this changed? There really should be a way to configure the health check path. Currently my root path redirects... – Jonathan Lin Apr 15 '20 at 06:30
This has been fixed very recently. Define a readinessProce on the Deployment. Recreate your Ingress. It will pick up the health check path from the readinessProbe. – Jonathan Lin Apr 15 '20 at 06:40

score 5 · Answer 3 · answered Apr 15 '20 at 06:45

5

This is now possible in the latest GKE (I am on 1.14.10-gke.27, not sure if that matters)

Define a readinessProbe on your container in your Deployment.
Recreate your Ingress.
The health check will point to the path in readinessProbe.httpGet.path of the Deployment yaml config.

answered Apr 15 '20 at 06:45

Jonathan Lin

19,922
7
69
65

thanks for mentioning: 2. Ingress recreation - this proved to be important. For **inferred** health checks, changing a readinessProbe after creation is not copied to the external LB. Not sure if the new CRD BackendConfig does it, but without recreation one has to edit the LB HCs directly. There's a Warning to that extent in the [docs](https://cloud.google.com/kubernetes-engine/docs/concepts/ingress#interpreted_hc) – wbob Jul 04 '21 at 13:10
You sir, you are a true hero – Nicolas de Lima Aug 25 '23 at 23:03

score 3 · Answer 4 · answered Aug 01 '20 at 18:52

Google has recently added support for CRD that can configure your Backend Services along with healthchecks:

apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: backend-config
  namespace: prod
spec:
  healthCheck:
    checkIntervalSec: 30
    port: 8080
    type: HTTP #case-sensitive
    requestPath: /healthcheck

See here.

score 0 · Answer 5 · answered Aug 01 '19 at 12:25

Another reason why Google Cloud Load Balancer does not pick-up GCE health check configuration from Kubernetes Pod readiness probe could be that the service is configured as "selectorless" (the selector attribute is empty and you manage endpoints directly).

This is the case with e.g. kube-lego: see https://github.com/jetstack/kube-lego/issues/68#issuecomment-303748457 and https://github.com/jetstack/kube-lego/issues/68#issuecomment-327457982.

Original question does have selector specified in the service, so this hint doesn't apply. This hints serves visitors that have the same problem with a different cause.

GCE Ingress not picking up health check from readiness probe

5 Answers5

Linked