1

I've got a spring-boot application deployed in kubernetes, with actuator on port 8081. The main application is working on port 8080. When I describe the instance, the checks are failing:

$ kubectl describe pod app-844d96f469-8vkbn
...
  Warning  Unhealthy  29s (x4 over 48s)  kubelet            Readiness probe failed: Get "http://192.168.13.189:8081/actuator/health/readiness": dial tcp 192.168.13.189:8081: connect: connection refused
  Warning  Unhealthy  29s (x2 over 39s)  kubelet            Liveness probe failed: Get "http://192.168.13.189:8081/actuator/health/liveness": dial tcp 192.168.13.189:8081: connect: connection refused

When I ssh into the pod, the checks show that the application is healthy, both on localhost and the ip addresses from the error message:

$ kubectl exec -it pod/app-844d96f469-8vkbn -- /bin/sh
# curl localhost:8081/actuator/health/liveness
{"status":"UP"}
# curl localhost:8081/actuator/health/readiness
{"status":"UP"}
# curl 192.168.13.189:8081/actuator/health/liveness
{"status":"UP"}
# curl 192.168.13.189:8081/actuator/health/readiness
{"status":"UP"}

The application was healthy within the 10*10=100 second window, so this isn't just an issue of adjusting the threshold and period.

This is the kubernetes config for the checks from the deployment:

        livenessProbe:
          httpGet:
            path: "/actuator/health/liveness"
            port: 8081
          failureThreshold: 10
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: "/actuator/health/readiness"
            port: 8081
          failureThreshold: 10
          periodSeconds: 10

I'd appreciate any help with this issue!

Edit: Adding the Dockerfile, since Thomas asked if 8081 was exposed on the image. I didn't have to do anything for port 8080 to be exposed, but maybe the question wasn't related to the Dockerfile.

FROM eclipse-temurin:17.0.5_8-jre
RUN mkdir /opt/app
ARG JAR_FILE
COPY ${JAR_FILE} /opt/app/app.jar
ENTRYPOINT ["java","-jar","/opt/app/app.jar"]
Kevin
  • 1,080
  • 3
  • 15
  • 41
  • Is the port 8081 exposed by the container image ? – Thomas Nov 14 '22 at 07:39
  • Where would I expose 8081? In the dockerfile? I'm not sure how I exposed 8080. – Kevin Nov 14 '22 at 17:27
  • What kubernetes distribution are you using? Do you use network policies? – Thomas Nov 14 '22 at 22:37
  • I'm using kubernetes on AWS, version 1.22. I do not see anything with `kind: NetworkPolicy` in my codebase. – Kevin Nov 15 '22 at 02:33
  • Did you try `kubectl port-forward` to check if you could visit specified port&path from outside the container? Does your java program start too slow, in which case you should define a `startupProbe`? – YwH Nov 23 '22 at 04:04
  • It's nothing to do with the container image, the pod yaml / service decides what ports are accessible where. there's a workaround where you can execute a script inside the container, prolly not exactly what you want but should work: https://stackoverflow.com/questions/62185723/liveness-probe-for-microservice-without-http/62186461#62186461 – stringy05 Nov 23 '22 at 22:19

1 Answers1

2

I suppose that your application takes too long to become ready before it fails due to liveness probe. Spring boot / Java applications do heavy initialization on startup and when CPU is limited, it is slow and takes a long time.

Here is a good explanation: https://faun.pub/java-application-optimization-on-kubernetes-on-the-example-of-a-spring-boot-microservice-cf3737a2219c

A simple service takes 80 seconds on 0.5 CPU while it only takes 12 seconds on 3 CPUs.

If you hava CPU limit set, just leave it out. Only set a request. And maybe also configure an initialDelaySeconds: 60 or so for the liveness probe.

user2311578
  • 798
  • 3
  • 7