k8s spring boot pod failing readiness and liveness probe

Question

I have configured a spring-boot pod and configured the liveness and readiness probes. When I start the pod, the describe command is showing the below output.

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  92s                default-scheduler  Successfully assigned pradeep-ns/order-microservice-rs-8tqrv to pool-h4jq5h014-ukl3l
  Normal   Pulled     43s (x2 over 91s)  kubelet            Container image "classpathio/order-microservice:latest" already present on machine
  Normal   Created    43s (x2 over 91s)  kubelet            Created container order-microservice
  Normal   Started    43s (x2 over 91s)  kubelet            Started container order-microservice
  Warning  Unhealthy  12s (x6 over 72s)  kubelet            Liveness probe failed: Get "http://10.244.0.206:8222/actuator/health/liveness": dial tcp 10.244.0.206:8222: connect: connection refused
  Normal   Killing    12s (x2 over 52s)  kubelet            Container order-microservice failed liveness probe, will be restarted
  Warning  Unhealthy  2s (x8 over 72s)   kubelet            Readiness probe failed: Get "http://10.244.0.206:8222/actuator/health/readiness": dial tcp 10.244.0.206:8222: connect: connection refused

The pod definition is like below

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: order-microservice-rs
  labels:
    app: order-microservice
spec:
  replicas: 1
  selector:
    matchLabels:
      app: order-microservice
  template:
    metadata:
      name: order-microservice
      labels:
        app: order-microservice
    spec:
      containers:
        - name: order-microservice
          image: classpathio/order-microservice:latest
          imagePullPolicy: IfNotPresent
          env:
            - name: SPRING_PROFILES_ACTIVE
              value: dev
            - name: SPRING_DATASOURCE_USERNAME
              valueFrom:
                secretKeyRef:
                  key: username
                  name: db-credentials
            - name: SPRING_DATASOURCE_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: db-credentials
          volumeMounts:
            - name: app-config
              mountPath: /app/config
            - name: app-logs
              mountPath: /var/log
          livenessProbe:
            httpGet:
              port: 8222
              path: /actuator/health/liveness
            initialDelaySeconds: 10
            periodSeconds: 10
          readinessProbe:
            httpGet:
              port: 8222
              path: /actuator/health/readiness
            initialDelaySeconds: 10
            periodSeconds: 10
          resources:
            requests:
              memory: "550Mi"
              cpu: "500m"
            limits:
              memory: "550Mi"
              cpu: "750m"
      volumes:
        - name: app-config
          configMap:
            name: order-microservice-config
        - name: app-logs
          emptyDir: {}
      restartPolicy: Always

If I disable the liveness and readiness probe in the replica-set manifest and I exec into the pod, I am getting a valid response when invoking http://localhost:8222/actuator/health/liveness and http://localhost:8222/actuator/health/readiness endpoint. Why is my pod restarting and failing when invoking the readiness and liveness endpoint with Kubernetes. Where am I going wrong?

Update If I remove the resource section, the pods are running but when added the resource parameters, the probes are failing.

when the pod restart, did the spring boot application start complete? @zilcuanu — Dolphin, Feb 02 '22 at 15:55

score 3 · Answer 1 · answered Feb 02 '22 at 15:41

3

When you limit the container / spring application to 0.5 cores (500 millicores) the startup probably takes longer than the given liveness probe thresholds.

You can either increase them, or use a startupProbe with more relaxed settings (f.e. failureThreshold 10). You can reduce the period for the liveness probe in that case and get faster feedback after a successful container start was detected.

answered Feb 02 '22 at 15:41

Thomas

11,272
2
24
40

How to know the cores and mem settings for spring boot applications? What is the guideline to arrive at the number – zilcuanu Feb 02 '22 at 15:47
That depends on the specific application. I recommend to observe it during startup and regular operation and derive appropriate values. – Thomas Feb 07 '22 at 14:23

score 2 · Answer 2 · answered Feb 02 '22 at 16:05

Your pod config only give 0.5 Core of CPU, and your check time was too short. The spring boot start may take a long time more than 10 seconds according your server CPU performance. This is my config of spring boot pod may give you a point.

"livenessProbe": {
              "httpGet": {
                "path": "/actuator/liveness",
                "port": 11032,
                "scheme": "HTTP"
              },
              "initialDelaySeconds": 90,
              "timeoutSeconds": 30,
              "periodSeconds": 30,
              "successThreshold": 1,
              "failureThreshold": 3
            },
            "readinessProbe": {
              "httpGet": {
                "path": "/actuator/health",
                "port": 11032,
                "scheme": "HTTP"
              },
              "initialDelaySeconds": 60,
              "timeoutSeconds": 30,
              "periodSeconds": 30,
              "successThreshold": 1,
              "failureThreshold": 3
            },

and I did not limit the CPU and memory resource, if you limit the CPU, it will take more time. Hop this could help you.

score 0 · Answer 3 · answered Feb 02 '22 at 16:27

When you are trying the request against your localhost, and it works, it is not a guarantee that it is going to work on other network interfaces. Kubelet is a node agent, so the request is going to your eth0, or equivalent, not your localhost.

You can check it by making the request from another pod to your pod's IP address, or the service backing it up.

Probably you are making your application to serve on localhost, while you have to make it serve on 0.0.0.0, or eth0.

k8s spring boot pod failing readiness and liveness probe

3 Answers3