Kubernetes pod gets recreated when deleted

Question

I have started pods with command

$ kubectl run busybox \
--image=busybox \
--restart=Never \
--tty \
-i \
--generator=run-pod/v1

Something went wrong, and now I can't delete this Pod.

I tried using the methods described below but the Pod keeps being recreated.

$ kubectl delete pods  busybox-na3tm
pod "busybox-na3tm" deleted

$ kubectl get pods
NAME                                     READY     STATUS              RESTARTS   AGE
busybox-vlzh3                            0/1       ContainerCreating   0          14s

$ kubectl delete pod busybox-vlzh3 --grace-period=0

$ kubectl delete pods --all
pod "busybox-131cq" deleted
pod "busybox-136x9" deleted
pod "busybox-13f8a" deleted
pod "busybox-13svg" deleted
pod "busybox-1465m" deleted
pod "busybox-14uz1" deleted
pod "busybox-15raj" deleted
pod "busybox-160to" deleted
pod "busybox-16191" deleted

$ kubectl get pods --all-namespaces
NAMESPACE   NAME            READY     STATUS              RESTARTS   AGE
default     busybox-c9rnx   0/1       RunContainerError   0          23s

Did you somehow manage to create a replication controller by passing wrong arguments. What do you get for ``kubectl get all -o name``? — Graham Dumpleton, Nov 18 '16 at 21:28
No, I had not created replication controller until today which I was able to delete it without any issue. This is just pod by itself I created two days ago. Oh wow I got 2599 pod/busybox # kubectl get all -o namepod/busybox-zzt7p ... # kubectl get all -o name | wc -l 2599 — yman, Nov 18 '16 at 21:39
Can you check `kubectl get events` to see what is creating these objects? — Anirudh Ramanathan, Nov 18 '16 at 21:59
try `kubctl get rc` to see if a ReplicationController was created. If so, delete that, then delete the pods. — MrE, Nov 18 '16 at 23:20
what version of kubernetes are you running? Depending on your kubernetes version it? It could behave differently. for example before 1.2 it always created deployment. `kubectl get deployment` — lwolf, Nov 19 '16 at 19:46
`# kubectl version` -> v1.2.0 `#kubectl get events` --> "Error syncing pod, skipping: failed to "StartContainer" for "busybox" with RunContainerError: "runContainer: API error (500): Container command not found or does not exist.\n" `# kubectl get rc` --> shows nothing `#kubectl get deployment` --> shows nothing ----- It looks like it got stuck in downloading image. I wonder how I can delete or stop this container. `#kubectl get pods` _NAME READY STATUS RESTARTS AGE_ _busybox-zehyn 0/1 ContainerCreating 0 8s_ — yman, Nov 29 '16 at 18:02
If someone ends by here:- Deleteing deployments solved the issue for me. `kubectl delete deployment `. To get the deployment name, do `kubectl get deployments` — Vasanth Sriram, Dec 07 '16 at 08:44

score 509 · Accepted Answer · edited Mar 23 '21 at 22:04

509

You need to delete the deployment, which should in turn delete the pods and the replica sets https://github.com/kubernetes/kubernetes/issues/24137

To list all deployments:

kubectl get deployments --all-namespaces

Then to delete the deployment:

kubectl delete -n NAMESPACE deployment DEPLOYMENT

Where NAMESPACE is the namespace it's in, and DEPLOYMENT is the name of the deployment. If NAMESPACE is default, leave off the -n option altogether.

In some cases it could also be running due to a job or daemonset. Check the following and run their appropriate delete command.

kubectl get jobs

kubectl get daemonsets.app --all-namespaces

kubectl get daemonsets.extensions --all-namespaces

edited Mar 23 '21 at 22:04

Noel Yap

18,822
21
92
144

answered May 30 '17 at 23:10

nomad

5,442
1
18
8

1

How do you bring the deployment back afterwards? – Jamey Jul 24 '18 at 23:38
2

@Jamey you create it again with the `kubectl create` command. – Illya Gerasymchuk Dec 26 '18 at 16:40
1

does not need to be a deployment. could be a job. so make sure to also check ```kubectl get jobs``` – bucky Jun 18 '19 at 08:41
2

To delete multiple object types, not just deployments, try: `kubectl delete replicasets,subscriptions,deployments,jobs,services,pods --all -n ` – Noam Manos Nov 20 '19 at 09:46
`kubectl delete deployments --namespace=default nginx-deployment` , this command worked for me – Asanka Siriwardena Jul 31 '22 at 10:47

score 57 · Answer 2 · edited Jan 21 '22 at 09:05

Instead of trying to figure out whether it is a deployment, deamonset, statefulset... or what (in my case it was a replication controller that kept spanning new pods :) In order to determine what it was that kept spanning up the image I got all the resources with this command:

kubectl get all

Of course you could also get all resources from all namespaces:

kubectl get all --all-namespaces

or define the namespace you would like to inspect:

kubectl get all -n NAMESPACE_NAME

Once I saw that the replication controller was responsible for my trouble I deleted it:

kubectl delete replicationcontroller/CONTROLLER_NAME

score 28 · Answer 3 · edited Jan 21 '22 at 04:38

28

If your pod has name like name-xxx-yyy, it could be controlled by a replicasets.apps named name-xxx, you should delete that replicaset first before deleting the pod:

kubectl delete replicasets.apps name-xxx

edited Jan 21 '22 at 04:38

Mike

1,080
1
9
25

answered Jul 12 '18 at 09:26

Hieu Vo

3,105
30
31

1

Thanks! For my case, it was a specific job that was recreating it. So: `kubectl delete --all jobs -n ` – yclian Oct 13 '18 at 09:51
Find the replica-set with `kubectl get replicasets.apps -n ` (or --all-namespaces) – Noam Manos Nov 19 '19 at 15:14

score 19 · Answer 4 · answered Mar 09 '21 at 03:08

19

Obviously something is respawning the pod. While a lot of the other answers have you looking at everything (replica sets, jobs, deployments, stateful sets, ...) to find what may be respawning the pod, you can instead just look at the pod to see what spawned it. For example do:

$ kubectl describe pod $mypod | grep 'Controlled By:'
Controlled By:  ReplicaSet/foobar

This tells you exactly what created the pod. You can then go and delete that.

answered Mar 09 '21 at 03:08

phemmer

6,882
3
33
31

This is an incredible answer and saved my "life" as we devs tend to say. Cheers – A Webb Mar 22 '22 at 02:29

score 15 · Answer 5 · answered Jan 31 '19 at 13:08

Look out for stateful sets as well

kubectl get sts --all-namespaces

to delete all the stateful sets in a namespace

kubectl --namespace <yournamespace> delete sts --all

to delete them one by one

kubectl --namespace ag1 delete sts mssql1 
kubectl --namespace ag1 delete sts mssql2
kubectl --namespace ag1 delete sts mssql3

score 10 · Answer 6 · edited Mar 01 '21 at 12:09

10

This will provide information about all the pods,deployments, services and jobs in the namespace.

kubectl get pods,services,deployments,jobs

pods can either be created by deployments or jobs

kubectl delete job [job_name]
kubectl delete deployment [deployment_name]

If you delete the deployment or job then restart of the pods can be stopped.

edited Mar 01 '21 at 12:09

damd

6,116
7
48
77

answered Mar 15 '19 at 23:03

Rohith

327
4
12

Noam Manos · Answer 7 · 2019-11-20T12:24:01.313

Many answers here tells to delete a specific k8s object, but you can delete multiple objects at once, instead of one by one:

kubectl delete deployments,jobs,services,pods --all -n <namespace>

In my case, I'm running OpenShift cluster with OLM - Operator Lifecycle Manager. OLM is the one who controls the deployment, so when I deleted the deployment, it was not sufficient to stop the pods from restarting.

Only when I deleted OLM and its subscription, the deployment, services and pods were gone.

First list all k8s objects in your namespace:

$ kubectl get all -n openshift-submariner

NAME                                       READY   STATUS    RESTARTS   AGE
pod/submariner-operator-847f545595-jwv27   1/1     Running   0          8d  
NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/submariner-operator-metrics   ClusterIP   101.34.190.249   <none>        8383/TCP   8d
NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/submariner-operator   1/1     1            1           8d
NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/submariner-operator-847f545595   1         1         1       8d

OLM is not listed with get all, so I search for it specifically:

$ kubectl get olm -n openshift-submariner

NAME                                                      AGE
operatorgroup.operators.coreos.com/openshift-submariner   8d
NAME                                                             DISPLAY      VERSION
clusterserviceversion.operators.coreos.com/submariner-operator   Submariner   0.0.1

Now delete all objects, including OLMs, subscriptions, deployments, replica-sets, etc:

$ kubectl delete olm,svc,rs,rc,subs,deploy,jobs,pods --all -n openshift-submariner

operatorgroup.operators.coreos.com "openshift-submariner" deleted
clusterserviceversion.operators.coreos.com "submariner-operator" deleted
deployment.extensions "submariner-operator" deleted
subscription.operators.coreos.com "submariner" deleted
service "submariner-operator-metrics" deleted
replicaset.extensions "submariner-operator-847f545595" deleted
pod "submariner-operator-847f545595-jwv27" deleted

List objects again - all gone:

$ kubectl get all -n openshift-submariner
No resources found.

$ kubectl get olm -n openshift-submariner
No resources found.

score 6 · Answer 8 · answered Feb 07 '19 at 00:48

After taking an interactive tutorial I ended up with a bunch of pods, services, deployments:

me@pooh ~ > kubectl get pods,services
NAME                                       READY   STATUS    RESTARTS   AGE
pod/kubernetes-bootcamp-5c69669756-lzft5   1/1     Running   0          43s
pod/kubernetes-bootcamp-5c69669756-n947m   1/1     Running   0          43s
pod/kubernetes-bootcamp-5c69669756-s2jhl   1/1     Running   0          43s
pod/kubernetes-bootcamp-5c69669756-v8vd4   1/1     Running   0          43s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   37s
me@pooh ~ > kubectl get deployments --all-namespaces
NAMESPACE     NAME                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
default       kubernetes-bootcamp   4         4         4            4           1h
docker        compose               1         1         1            1           1d
docker        compose-api           1         1         1            1           1d
kube-system   kube-dns              1         1         1            1           1d

To clean up everything, delete --all worked fine:

me@pooh ~ > kubectl delete pods,services,deployments --all
pod "kubernetes-bootcamp-5c69669756-lzft5" deleted
pod "kubernetes-bootcamp-5c69669756-n947m" deleted
pod "kubernetes-bootcamp-5c69669756-s2jhl" deleted
pod "kubernetes-bootcamp-5c69669756-v8vd4" deleted
service "kubernetes" deleted
deployment.extensions "kubernetes-bootcamp" deleted

That left me with (what I think is) an empty Kubernetes cluster:

me@pooh ~ > kubectl get pods,services,deployments
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   8m

Jafari Shahabaj · Answer 9 · 2022-09-10T12:30:25.273

5

Firstly list the deployments

kubectl get deployments

After that delete the deployment

kubectl delete deployment <deployment_name>

edited Sep 10 '22 at 12:30

answered Sep 10 '22 at 12:29

Jafari Shahabaj

51
1
2

Without setting context or specifying namespace name in the commands, they will return resource information in default namespace. Also the original poster has specified that the pods we’re getting created by K8S job not deployments – Nataraj Medayhal Sep 14 '22 at 14:47

score 4 · Answer 10 · answered Mar 14 '18 at 00:27

4

In some cases the pods will still not go away even when deleting the deployment. In that case to force delete them you can run the below command.

kubectl delete pods podname --grace-period=0 --force

answered Mar 14 '18 at 00:27

emirhosseini

545
2
8
17

2

This won't resolve the problem when the pod created by deployment, jobs or any other kind of controllers if the strategy type is set to `Recreate`. – S.K. Venkat Aug 27 '19 at 05:28
The scenario being described is due to the continued existence of the deployment. So the solution is to delete the deployment. The edge-case you are referring to is not the answer and as-such needs more description as to *when* this is necessary and what causes it to be necessary. – Scala Enthusiast Jul 18 '23 at 14:26

score 4 · Answer 11 · answered Mar 14 '18 at 01:56

When the pod is recreating automatically even after the deletion of the pod manually, then those pods have been created using the Deployment. When you create a deployment, it automatically creates ReplicaSet and Pods. Depending upon how many replicas of your pod you mentioned in the deployment script, it will create those number of pods initially. When you try to delete any pod manually, it will automatically create those pod again.

Yes, sometimes you need to delete the pods with force. But in this case force command doesn’t work.

I get a warning when I try this that the pod may live on as a zombie process so wasn't what I wanted.. — Chanoch, Feb 20 '19 at 13:02

score 4 · Answer 12 · answered Jan 07 '19 at 17:10

4

Instead of removing NS you can try removing replicaSet

kubectl get rs --all-namespaces

Then delete the replicaSet

kubectl delete rs your_app_name

answered Jan 07 '19 at 17:10

Vadim Sluzky

259
1
2

S.K. Venkat · Answer 13 · 2019-08-27T05:31:36.347

The root cause for the question asked was the deployment/job/replicasets spec attribute strategy->type which defines what should happen when the pod will be destroyed (either implicitly or explicitly). In my case, it was Recreate.

As per @nomad's answer, deleting the deployment/job/replicasets is the simple fix to avoid experimenting with deadly combos before messing up the cluster as a novice user.

Try the following commands to understand the behind the scene actions before jumping into debugging :

kubectl get all -A -o name
kubectl get events -A | grep <pod-name>

score 2 · Answer 14 · answered Nov 15 '19 at 12:57

2

In my case I deployed via a YAML file like kubectl apply -f deployment.yaml and the solution appears to be to delete via kubectl delete -f deployment.yaml

answered Nov 15 '19 at 12:57

Chris_Rands

38,994
14
83
119

score 2 · Answer 15 · answered Jun 16 '22 at 06:42

Kubernetes always works in the format like:

deployments >>> replicasets >>> pods

first edit deployment with 0 replicas and then scale deployment with desired replicas(run below command).You will see new replicaset has been created and pods will also run with desired count.

*

IN-Linux:~ anuragmanikkame$ kubectl scale deploy tomcat -n dev-namespace --replicas=2 deployment.extensions/tomcat scaled

score 1 · Answer 16 · answered Feb 07 '19 at 14:14

1

If you have a job that continues running, you need to search the job and delete it:

kubectl get job --all-namespaces | grep <name>

and

kubectl delete job <job-name>

answered Feb 07 '19 at 14:14

kurkop

520
4
11

score 1 · Answer 17 · answered Jun 18 '19 at 11:32

1

You can do kubectl get replicasets check for old deployment based on age or time

Delete old deployment based on time if you want to delete same current running pod of application

kubectl delete replicasets <Name of replicaset>

answered Jun 18 '19 at 11:32

Harsh Manvar

27,020
6
48
102

score 1 · Answer 18 · edited Aug 20 '19 at 10:11

1

I also faced the issue, I have used below command to delete deployment.

kubectl delete deployments DEPLOYMENT_NAME

but still pods was recreating, So I crossed check the Replica Set by using below command

kubectl get rs

then edit the replicaset to 1 to 0

kubectl edit rs REPICASET_NAME

edited Aug 20 '19 at 10:11

Andronicus

25,419
17
47
88

answered Aug 20 '19 at 09:46

user2688181

19
1
6

score 1 · Answer 19 · answered Jun 20 '20 at 15:31

With deployments that have stateful sets (or services, jobs, etc.) you can use this command:

This command terminates anything that runs in the specified <NAMESPACE>

kubectl -n <NAMESPACE> delete replicasets,deployments,jobs,service,pods,statefulsets --all

And forceful

kubectl -n <NAMESPACE> delete replicasets,deployments,jobs,service,pods,statefulsets --all --cascade=true --grace-period=0 --force

score 1 · Answer 20 · answered Oct 04 '21 at 10:58

There is basically two ways to remove PODS

kubectl scale --replicas=0 deploy name_of_deployment. This will set the number of replica to 0 and hence it will not restart the pods again.
Use helm to uninstall the chart which you have implemented in your pipeline. Do not delete the deployment directly, instead use helm to uninstall the chart which will remove all objects it created.

score 1 · Answer 21 · edited Feb 17 '22 at 12:36

1

The fastest solution for me was installing Lens IDE and removing the service under de DEPLOYMENTS tab. Just delete from this tab and the replica will be deleted too.

Best regards

edited Feb 17 '22 at 12:36

Dharman

30,962
25
85
135

answered Feb 17 '22 at 12:30

Rizzle Sizzle

11
3

score 0 · Answer 22 · answered Mar 23 '20 at 11:54

I experienced a similar problem: after deleting the deployment (kubectl delete deploy <name>), the pods kept "Running" and where automatically re-created after deletion (kubectl delete po <name>).

It turned out that the associated replica set was not deleted automatically for some reason, and after deleting that (kubectl delete rs <name>), it was possible to delete the pods.

score 0 · Answer 23 · edited May 17 '21 at 06:54

This has happened to me with some broken 'helm' installs. You might have a bit of a messed up deployment. If none of the previous suggestions work, look for a daemonset and delete that.

eg kubectl get daemonset --namespace

then delete daemonset

kubectl delete daemonset --namespace <NAMESPACE> --all --force

then try to delete the pods.

kubectl delete pod --namespace  <NAMESPACE> --all --force

Check if pods are gone.

kubectl get pods --all-namespaces

score 0 · Answer 24 · edited Dec 13 '22 at 05:12

0

In my case I use these below

kubectl get all --all-namespaces 
kubectl delete deployment statefulset-deploymentnament(choose your deployment name)
kubectl delete sts -n default(choose your namespace) --all 
kubectl get pods --all-namespaces

Problem got resolved

edited Dec 13 '22 at 05:12

Anakin

1,889
1
13
27

answered Dec 08 '22 at 12:11

user20723420

1
1

Kubernetes pod gets recreated when deleted

24 Answers24

Linked