Reasons of Pod Status Failed

Question

If Pod's status is Failed, Kubernetes will try to create new Pods until it reaches terminated-pod-gc-threshold in kube-controller-manager. This will leave many Failed Pods in a cluster and need to be cleaned up.

Are there other reasons except Evicted that will cause Pod Failed?

What does `kubectl describe pod NAME` say? Please add the output to your question. Are the health checks for the pod configured correctly? Can you show their configuration? — ProgrammingLlama, Apr 08 '19 at 09:28
@John I haven't seen other reasons except `Evicted` that cause Pod `Failed` and want to know if there're other reaons :D — flyer, Apr 08 '19 at 09:37
Yes, I want to know if anyone has seen reasons other than `Evicted` that cause Pod `Failed`. — flyer, Apr 08 '19 at 10:01

Arbaaz · Answer 1 · 2020-11-01T06:49:59.953

There can be many causes for the POD status to be FAILED. You just need to check for problems(if there exists any) by running the command

kubectl -n <namespace> describe pod <pod-name>

Carefully check the EVENTS section where all the events those occurred during POD creation are listed. Hopefully you can pinpoint the cause of failure from there.

However there are several reasons for POD failure, some of them are the following:

Wrong image used for POD.
Wrong command/arguments are passed to the POD.
Kubelet failed to check POD liveliness(i.e., liveliness probe failed).
POD failed health check.
Problem in network CNI plugin (misconfiguration of CNI plugin used for networking).

For example:

In the above example, the image "not-so-busybox" couldn't be pulled as it doesn't exist, so the pod FAILED to run. The pod status and events clearly describe the problem.

score 3 · Answer 2 · answered Jul 07 '20 at 16:59

3

Simply do this:

kubectl get pods <pod_name> -o yaml

And in the output, towards the end, you can see something like this:

This will give you a good idea of where exactly did the pod fail and what happened.

answered Jul 07 '20 at 16:59

Aashish Chaubey

589
1
8
22

score 2 · Answer 3 · edited Dec 20 '22 at 00:05

PODs will not survive scheduling failures, node failures, or other evictions, such as lack of resources, or in the case of node maintenance. Pods should not be created manually but almost always via controllers like Deployments (self-healing, replication etc).

Reason why pod failed or was terminated can be obtain by

kubectl describe pod <pod_name>

Others situation I have encountered when pod Failed:

Issues with image (not existing anymore)
When pod is attempting to access i.e ConfigMap or Secrets but it is not found in namespace.
Liveness Probe Failure
Persistent Volume fails to mount
Validation Error

In addition, eviction is based on resources - EvictionPolicy

It can be also caused by DRAINing the Node/Pod. You can read about DRAIN here.

Reasons of Pod Status Failed

3 Answers3