Pods stuck in Terminating status

Question

I tried to delete a ReplicationController with 12 pods and I could see that some of the pods are stuck in Terminating status.

My Kubernetes cluster consists of one control plane node and three worker nodes installed on Ubuntu virtual machines.

What could be the reason for this issue?

NAME        READY     STATUS        RESTARTS   AGE
pod-186o2   1/1       Terminating   0          2h
pod-4b6qc   1/1       Terminating   0          2h
pod-8xl86   1/1       Terminating   0          1h
pod-d6htc   1/1       Terminating   0          1h
pod-vlzov   1/1       Terminating   0          1h

Answer 1

You can use following command to delete the POD forcefully.

kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>

Answer 2

The original question is "What could be the reason for this issue?" and the answer is discussed at https://github.com/kubernetes/kubernetes/issues/51835 & https://github.com/kubernetes/kubernetes/issues/65569 & see https://www.bountysource.com/issues/33241128-unable-to-remove-a-stopped-container-device-or-resource-busy

Its caused by docker mount leaking into some other namespace.

You can logon to pod host to investigate.

minikube ssh
docker container ps | grep <id>
docker container stop <id> 

Answer 3

Force delete the pod:

kubectl delete pod --grace-period=0 --force --namespace <NAMESPACE> <PODNAME>

The --force flag is mandatory.

Answer 4

I found this command more straightforward:

for p in $(kubectl get pods | grep Terminating | awk '{print $1}'); do kubectl delete pod $p --grace-period=0 --force;done

It will delete all pods in Terminating status in default namespace.

Answer 5

In my case the --force option didn't quite work. I could still see the pod ! It was stuck in Terminating/Unknown mode. So after running

kubectl -n redis delete pods <pod> --grace-period=0 --force

I ran

kubectl -n redis patch pod <pod> -p '{"metadata":{"finalizers":null}}'

Answer 6

Delete the finalizers block from resource (pod,deployment,ds etc...) yaml:

"finalizers": [
  "foregroundDeletion"
]

Answer 7

I stumbled upon this recently to free up resource in my cluster. here is the command to delete them all.

kubectl get pods --all-namespaces | grep Terminating | while read line; do
  pod_name=$(echo $line | awk '{print $2}' ) \
  name_space=$(echo $line | awk '{print $1}' ); \
  kubectl delete pods $pod_name -n $name_space --grace-period=0 --force
done

hope this help someone who read this

Answer 8

Practical answer -- you can always delete a terminating pod by running:

kubectl delete pod NAME --grace-period=0

Historical answer -- There was an issue in version 1.1 where sometimes pods get stranded in the Terminating state if their nodes are uncleanly removed from the cluster.

Answer 9

Please try below command:

kubectl patch pod <pod>-p '{"metadata":{"finalizers":null}}'

Answer 10

For my case, I don't like workaround. So there are steps :

• k get pod -o wide -> this will show which Node is running the pod
• k get nodes -> Check status of that node... I got it NotReady

I went and I fixed that node. For my case, it's just restart kubelet :

• ssh that-node -> run swapoff -a && systemctl restart kubelet (Or systemctl restart k3s in case of k3s | or systemctl restart crio in other cases like OCP 4.x (k8s <1.23) )

Now deletion of pod should work without forcing the Poor pod.

Answer 11

If --grace-period=0 is not working then you can do:

kubectl delete pods <pod> --grace-period=0 --force

Answer 12

Force delete ALL pods in namespace:

kubectl delete pods --all -n <namespace> --grace-period 0 --force

Answer 13

I stumbled upon this recently when removing rook ceph namespace - it got stuck in Terminating state.

The only thing that helped was removing kubernetes finalizer by directly calling k8s api with curl as suggested here.

• kubectl get namespace rook-ceph -o json > tmp.json
• delete kubernetes finalizer in tmp.json (leave empty array "finalizers": [])
• run kubectl proxy in another terminal for auth purposes and run following curl request to returned port
• curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json 127.0.0.1:8001/k8s/clusters/c-mzplp/api/v1/namespaces/rook-ceph/finalize
• namespace is gone

Detailed rook ceph teardown here.

Answer 14

I used this command to delete the pods

kubectl delete pod --grace-period=0 --force --namespace <NAMESPACE> <PODNAME>

But when I tried run another pod, it didn't work, it was stuck in "Pending" state, it looks like the node itself was stuck.

For me, the solution was to recreate the node. I simply went to GKE console and deleted the node from the cluster and so GKE started another.

After that, everything started to work normally again.

Answer 15

I had to same issue in a production Kubernetes cluster.

A pod was stuck in Terminating phase for a while:

pod-issuing   mypod-issuing-0   1/1     Terminating   0  27h

I tried checking the logs and events using the command:

kubectl describe pod mypod-issuing-0 --namespace pod-issuing
kubectl logs mypod-issuing-0 --namespace pod-issuing

but none was available to view

How I fixed it:

I ran the command below to forcefully delete the pod:

kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>

This deleted the pod immediately and started creating a new one. However, I ran into the error below when another pod was being created:

Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data mypod-issuing-token-5swgg aws-iam-token]: timed out waiting for the condition

I had to wait for 7 to 10 minutes for the volume to become detached from the previous pod I deleted so that it can become available for this new pod I was creating.

Answer 16

you can use awk :

kubectl get pods --all-namespaces | awk '{if ($4=="Terminating") print "oc delete pod " $2 " -n " $1 " --force --grace-period=0 ";}' | sh

Answer 17

Before doing a force deletion i would first do some checks. 1- node state: get the node name where your node is running, you can see this with the following command:

"kubectl -n YOUR_NAMESPACE describe pod YOUR_PODNAME"

Under the "Node" label you will see the node name. With that you can do:

kubectl describe node NODE_NAME

Check the "conditions" field if you see anything strange. If this is fine then you can move to the step, redo:

"kubectl -n YOUR_NAMESPACE describe pod YOUR_PODNAME"

Check the reason why it is hanging, you can find this under the "Events" section. I say this because you might need to take preliminary actions before force deleting the pod, force deleting the pod only deletes the pod itself not the underlying resource (a stuck docker container for example).

Answer 18

I'd not recommend force deleting pods unless container already exited.

1. Verify kubelet logs to see what is causing the issue "journalctl -u kubelet"
2. Verify docker logs: journalctl -u docker.service
3. Check if pod's volume mount points still exist and if anyone holds lock on it.
4. Verify if host is out of memory or disk

Answer 19

I am going to try the most extense answer, because none of the above are wrong, but they do not work in all case scenarios.

The usual way to put an end to a terminating pod is:

kubectl delete pod -n ${namespace} ${pod} --grace-period=0 

But you may need to remove finalizers that could be preventing the POD from stoppoing using:

kubectl -n ${namespace} patch pod ${pod} -p '{"metadata":{"finalizers":null}}'

If none of that works, you can remove the pod from etcd with etcdctl:

# Define variables
ETCDCTL_API=3
certs-path=${HOME}/.certs/e
etcd-cert-path=${certs-path}/etcd.crt
etcd-key-path=${certs-path}/etcd.key
etcd-cacert-path=${certs-path}/etcd.ca
etcd-endpoints=https://127.0.0.1:2379
namespace=myns
pod=mypod

# Call etcdctl to remove the pod
etcdctl del \
--endpoints=${etcd-endpoints}\
--cert ${etcd-cert-path} \
--key ${etcd-client-key}\
--cacert ${etcd-cacert-path} \
--prefix \
/registry/pods/${namespace}/${pod} 

This last case should be used as last resource, in my case I ended having to do it due to a deadlock that prevented calico from starting in the node due to Pods under terminating status. Those pods won't be removed until calico is up, but they have reserved enough CPU to avoid calico, or any other pod, from Initializing.

Answer 20

To delete all pods in "Terminating" state in all namespaces:

kubectl get pods --all-namespaces | awk '/Terminating/{print $1 " " $2}' | while read -r namespace pod; do kubectl delete pod "$pod" -n "$namespace" --grace-period=0 --force;done

Answer 21

One reason WHY this happens can be turning off a node (without draining it). Fix in this case is to turn on the node again; then termination should succeed.

Answer 22

Following command with awk and xargs can be used along with --grace-period=0 --force to delete all the Pods in Terminating state.

kubectl get pods|grep -i terminating | awk '{print $1}' | xargs kubectl delete --grace-period=0 --force pod

Answer 23

go templates will work without awk, for me it works without --grace-period=0 --force but, add it if you like

this will output the command to delete the Terminated pods.

kubectl get pods --all-namespaces -otemplate='{{ range .items }}{{ if eq .status.reason  "Terminated" }}{{printf "kubectl delete pod -n %v %v\n" .metadata.namespace .metadata.name}}{{end}}{{end}}'

if you are happy with the output, you cat add | sh - to execute it. as follow:

kubectl get pods --all-namespaces -otemplate='{{ range .items }}{{ if eq .status.reason  "Terminated" }}{{printf "kubectl delete pod -n %v %v\n" .metadata.namespace .metadata.name}}{{end}}{{end}}' |sh -

Answer 24

My pods stuck in 'Terminating', even after I tried to restart docker & restart server. Resolved after edit the pod & delete items below 'finalizer'

$ kubectl -n mynamespace edit pod/my-pod-name

Answer 25

for me below command has resolved the issue

oc patch pvc pvc_name -p '{"metadata":{"finalizers":null}}