4

Issue:

We have added health check configuration to our function. However pod becomes Unhealthy due to timeout error in liveness and readiness checks and consequently getting restarted. However if I hit same health check url using CURL or browser it returns correct response.

Health check configuration reference.

We are using Kubernetes HPAv2 for auto-scaling (Reference).

test-function.yml

  test-function:
    lang: quarkus-java-with-fonts
    handler: ./test-function
    image: repo.azurecr.io/test-function:0.1
    labels:
      agentpool: openfaas
      com.openfaas.scale.min: "2"
      com.openfaas.scale.max: "10"
      com.openfaas.scale.factor: 0
    annotations:
     com.openfaas.health.http.path: "/health"
     com.openfaas.health.http.initialDelay: "30s"
    environment:
        secret_name: environment-variables
    secrets:
        - environment-variables 
    constraints:
        - agentpool=openfaas
    limits:
      cpu: 1500m
      memory: 1Gi
    requests:
      cpu: 500m
      memory: 500Mi

Error Trace :

Liveness probe failed: Get "http://XX.XXX.XX.XX:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Readiness probe failed: Get "http://XX.XXX.XX.XX:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Any idea what could be wrong.

Gabriel Robledo Ahumada
  • 1,542
  • 2
  • 14
Sandeep
  • 71
  • 1
  • 1
  • 3
  • Mikołaj Głodziak has already provided a valid answer. Was it useful ? If yes, Please upvote or accept the answer for greater visibility to community members. – Ramesh kollisetty Feb 28 '22 at 07:23

1 Answers1

1

These errors:

Liveness probe failed: Get "http://XX.XXX.XX.XX:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Readiness probe failed: Get "http://XX.XXX.XX.XX:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

mean that the HTTP request has failed. For the readiness and liveness probe to work properly, this type of request must be successful.

To find out where the problem is, you need to get the pod IP address. Run:

kubectl get pods -o wide

You should see an output similar to this:

NAME                        READY   STATUS    RESTARTS   AGE   IP          NODE                                       NOMINATED NODE   READINESS GATES
<my-pod-name>               1/1     Running   0          25d   10.92.3.4   <my-node-name>                             <none>           1/1

Take your IP and run:

kubectl exec -t <another_pod> -- curl -I <pod's cluster IP>

If you get a 200 response code, it means the endpoint is properly created and configured. Any other answer suggests there is a problem with your image.

See also:

Mikołaj Głodziak
  • 4,775
  • 7
  • 28
  • Hi @Mikołaj Głodziak - I am getting 200 response using kubectl exec. I have gone through other links that you have provided it suggests to use exec liveness probe instead of http liveness probe. However exec probe has its own downside as well. I am trying out exec probe currently. Thanks for your help. – Sandeep Aug 31 '21 at 06:33
  • Hi Mikolaj, Any idea how we can enable exec liveness probe using OpenFaaS. I have tried to go through OpenFaaS documentation but couldn't spot it. – Sandeep Sep 02 '21 at 04:25
  • Take in to [this doc](https://docs.openfaas.com/reference/workloads/). You can enable httpProbe in the helm chart and implement /_/health as a HTTP endpoint. – Mikołaj Głodziak Sep 02 '21 at 07:27
  • Does it enable exec liveness probe? I have gone through other links that you have provided it suggests to use exec liveness probe instead of http liveness probe. – Sandeep Sep 02 '21 at 12:14
  • Exec probes are default. [reference](https://docs.openfaas.com/architecture/production/#configure-function-health-check-probes). If you want to switch from "exec" liveness and readiness probes to httpProbes then use `--set faasnetes.httpProbe=true`, this can only be used with `--set operator.create=false`. Read more [here](https://artifacthub.io/packages/helm/openfaas/openfaas/3.2.3) – Mikołaj Głodziak Sep 06 '21 at 07:42
  • cURL may not exist everywhere, `ping` might be a good alternative in that case instead of `curl -I` – Arsham Arya Jul 19 '23 at 09:13