How to list Kubernetes recently deleted pods?

Question

Is there a way to get some details about Kubernetes pod that was deleted (stopped, replaced by new version).

I am investigating bug. I have logs with my pod name. That pod does not exist anymore, it was replaced by another one (with different configuration). New pod resides in same namespace, replication controller and service as old one.

Commands like

kubectl  get pods
kubectl  get pod <pod-name>

work only with current pods (live or stopped).

How I could get more details about old pods? I would like to see

when they were created
which environment variables they had when created
why and when they were stopped

iomv · Accepted Answer · 2019-08-02T13:35:01.377

67

As of today, kubectl get pods -a is deprecated, and as a result you cannot get deleted pods.

What you can do though, is to get a list of recently deleted pod names - up to 1 hour in the past unless you changed the ttl for kubernetes events - by running:

kubectl get event -o custom-columns=NAME:.metadata.name | cut -d "." -f1

You can then investigate further issues within your logging pipeline if you have one in place.

edited Aug 02 '19 at 13:35

answered Aug 01 '19 at 12:52

iomv

2,409
18
28

1

I'm not sure why there are so many unhelpful answers here, the real answer is the last sentence: "You can then investigate further issues within your logging pipeline if you have one in place." – user64204 Jan 31 '22 at 12:05

score 5 · Answer 2 · edited Jan 16 '23 at 22:32

As far as I know you cannot get the Pod details once the Pod is deleted. Can I know what is the usecase?

Example:

if a Pod is created using kubectl run busybox-test-pod-status --image=busybox --restart=Never -- /bin/false you will have a Pod with status terminated:error
if a Pod is created using kubectl run busybox-test-pod-status --image=busybox --restart=Never -- /bin/true you will have a Pod with status terminated:Completed
if a container in a Pod restarts: the Pod will be alive and you can get the logs of previous container (only the previous container) using kubectl logs --container <container name> --previous=true <pod name>
if you doing an upgrade of you app and you are creating Pods using Deployments. If the update deployment "say a new image", the Pod will be terminated and new Pod will be created. You can get the Pod details from the Deployment's YAML. if you want to get details of previous Pod you have see "spec" section of previous Deployment's YAML

score 0 · Answer 3 · answered Nov 16 '16 at 16:37

0

You can try kubectl logs --previous to list the logs of a previously stopped pod

http://kubernetes.io/docs/user-guide/kubectl/kubectl_logs/

You may also want to check out these debugging tips http://kubernetes.io/docs/user-guide/debugging-pods-and-replication-controllers/

answered Nov 16 '16 at 16:37

Matt Rickard

1,998
1
12
9

2

This command no longer seems to work, and a pod name is requested. – Eric Walker Feb 12 '18 at 20:19
12

It works, but it is meant to show logs of previous *containers* inside an existing pod. From the doc: "If true, print the logs for the previous instance of the container in a pod if it exists." – dvdgsng Feb 16 '18 at 10:54
6

This only works if the pod still exists, which is not what the OP asked. – Andrés Mejía Dec 07 '20 at 18:56

score 0 · Answer 4 · edited Sep 25 '19 at 10:57

0

There is a way to find out why pods were deleted and who deleted them. The only way to find out something is to set the ttl for k8s to be greater than the default 1h and search through the events:

kubectl get event -o custom-columns=NAME:.metadata.name | cut -d "." -f1

edited Sep 25 '19 at 10:57

41 72 6c

1,600
5
19
30

answered Sep 25 '19 at 07:55

Paul Chibulcuteanu

184
1
9

The only robust way of monitoring k8s objects is setting up an audit policy for monitoring k8s events. See official k8s documentation on it: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/ – Paul Chibulcuteanu Sep 26 '19 at 09:06
2

How is this different from the accepted answer? – iomv Sep 16 '21 at 10:39

score -4 · Answer 5 · answered Sep 13 '19 at 02:57

-4

If your container has previously crashed, you can access the previous container’s crash log with:

kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME}

answered Sep 13 '19 at 02:57

Sunil Gajula

1,117
2
14
30

4

This is not what the OP is asking. `kubectl logs --previous` only works if the pod still exists. Once the pod is deleted (after running `kubectl pod delete `, for example), this command no longer works. – Andrés Mejía Dec 07 '20 at 18:59

score -5 · Answer 6 · answered Nov 16 '16 at 16:53

-5

There is this flag:

-a, --show-all=false: When printing, show all resources (default hide terminated pods.)

But this may not help in all cases of old pods.

answered Nov 16 '16 at 16:53

manojlds

290,304
63
469
417

10

I haven't found this to work even with recently terminated pods. – Pixel Elephant Nov 16 '16 at 19:03
1

Yeah, seems like a bug or regression. – manojlds Nov 17 '16 at 06:14
4

I'm getting `Flag --show-all has been deprecated, will be removed in an upcoming release` with version 1.10 (client) and 1.9.2 (server) – gdvalderrama Apr 27 '18 at 10:17

score -9 · Answer 7 · edited Sep 06 '18 at 10:14

-9

kubectl get pods -a

you will get the list of running pods and the terminated pods in case you are searching for this

edited Sep 06 '18 at 10:14

oLen

5,177
1
32
48

answered Jan 30 '17 at 13:30

Hossam Khalil

17
3

This doesn't work for pods that have been terminated for a while – s g Dec 29 '17 at 17:47
This is now deprecated – Kieren Johnstone Mar 28 '19 at 07:42
2

This is now gone – Novaterata Apr 04 '19 at 18:42
@Novaterata & Kieren, could you provide a source for this? It seems to still exist as per: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get – Aaron N. Brock Apr 10 '19 at 16:59
@AaronN.Brock So it's actually the default now apparently. https://github.com/kubernetes/kubernetes/issues/67895 https://github.com/kubernetes/kubernetes/pull/69255 – Novaterata Apr 10 '19 at 18:19

Hushen · Answer 8 · 2020-04-27T22:40:02.573

If you want to see all the previously deleted pods and you are trying to fetch the previous pods.

Command line: kubectl get pods

in which you will get all the pod details, because every service has one or more pods and they have unique ip address

Here you can check the lifecycle of pods and what phases of pod has. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle

and you can see the previous pod logs by typing a command: kubectl logs --previous

How to list Kubernetes recently deleted pods?

8 Answers8

Linked