2

My kubernetes cluster has a pod deployment definition to have 10 pod copies. I ran into an issue where a pod instance's container will go into an unrecoverable state where the only mitigation I found is to restart the pod deployment through the following command:

kubectl rollout restart deployment my-pod

I would like to a way to auto execute a pod deployment restart through liveness probes or even pod condition status.

Are there any current out of the box solutions for this?

HDev
  • 29
  • 3
  • 3
  • Why is the state unrecoverable? Would a simple container restart not help? – Thomas Mar 16 '21 at 10:53
  • We are using an open source authentication which lists pods and acts as a middleman to perform auth for all of our pods. The middleman service can get in a bad state with listing pods and thus we need to restart a pod deployment. – HDev Mar 16 '21 at 17:28
  • Can you explain which tool you use? – Thomas Mar 16 '21 at 17:54
  • This service: https://github.com/Azure/aad-pod-identity – HDev Mar 16 '21 at 20:19

1 Answers1

0

Kubernetes's pod object provides you with a health check function. If the pod is unhealthy, the pod controller will automatically restart your pod.

Please check the official documentation here, https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/.

It provides all the implementations you will need in order to make the health check probe work.

Ryan Siu
  • 944
  • 4
  • 11
  • 2
    This is not accurate: The container will be restarted, not the pod, and it is handled by kubelet not a pod controller. – Thomas Mar 16 '21 at 10:54
  • 1
    Hi Ryan! Yep I understand the liveness probes. Although, as Thomas mentioned, this is a container restart, not a pod. Still looking for a solution that would restart a pod. – HDev Mar 16 '21 at 17:26
  • Did you get the solution for this? – Sadhvik Chirunomula Oct 31 '22 at 13:29