13

I have a use case to gracefully terminate the container where i have a script to kill the process gracefully from within the container by using the command "kill PID".( Which will send the TERM signal ) But I have liveness probe configured as well. Currently liveness probe is configured to probe at 60 second interval. So if the liveness probe take place shortly after the graceful termination signal is sent, the overall health of the container might become CRITICAL when the termination is still in progress. In this case the liveness probe will fail and container will be terminated immediately.

So i wanted to know whether kubelet kills the container with TERM or KILL.

Appreciate your support Thanks in advance

Beatrix Kiddo
  • 199
  • 1
  • 2
  • 10

1 Answers1

11

In Kubernetes, Liveness Probe checks for the health state of a container.

To answer your question on whether it uses SIGKILL or SIGTERM, the answer is both are used but in order. So here is what happens under the hood.

  1. Liveness probe check fails
  2. Kubernetes stops routing of traffic to the container
  3. Kubernetes restarts the container
  4. Kubernetes starts routing traffic to the container again

For container restart, SIGTERM is first sent with waits for a parameterized grace period, and then Kubernetes sends SIGKILL.

A hack around your issue is to use the attribute:

timeoutSeconds

This specifies how long a request can take to respond before it’s considered a failure. You can add and adjust this parameter if the time taken for your application to come online is predictable.

Also, you can play with readinessProbe before livenessProbe with an adequate delay for the container to come into service after restarting the process. Check https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/ for more details on which parameters to use.

Aby Sheffer
  • 483
  • 3
  • 4
  • Do you have a source for this information? I'm seeing this in my logs, and am suspicious about the word "kill": `Killing container with id docker://api:Container failed liveness probe.. Container will be killed and recreated.` – David Nov 11 '19 at 16:45
  • Kubernetes documentation only use the word "kill", BUT DONT SPECIFY IF THIS IS graceful or directly abrupt termination (common sense for a failed probe situation ?). Experimental tests seems to show that graceful termination is done, but I remark that this is not documented: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes – eramos Aug 18 '20 at 11:47