47

I am running selenium hubs and my pods are getting terminated frequently. I would like to look at the logs of the pods which are terminated. How to do it?

NAME                                               READY     STATUS              RESTARTS   AGE
chrome-75-0-0e5d3b3d-3580-49d1-bc25-3296fdb52666   0/2       Terminating         0          49s
chrome-75-0-29bea6df-1b1a-458c-ad10-701fe44bb478   0/2       Terminating         0          23s
chrome-75-0-8929d8c8-1f7b-4eba-96f2-918f7a0d77f5   0/2       ContainerCreating   0          7s

kubectl logs chrome-75-0-8929d8c8-1f7b-4eba-96f2-918f7a0d77f5
Error from server (NotFound): pods "chrome-75-0-8929d8c8-1f7b-4eba-96f2-918f7a0d77f5" not found
$ kubectl logs chrome-75-0-8929d8c8-1f7b-4eba-96f2-918f7a0d77f5 --previous
Error from server (NotFound): pods "chrome-75-0-8929d8c8-1f7b-4eba-96f2-918f7a0d77f5" not found
Jonathan Hall
  • 75,165
  • 16
  • 143
  • 189
SunilS
  • 2,030
  • 5
  • 34
  • 62

6 Answers6

22

Running kubectl logs -p will fetch logs from existing resources at API level. This means that terminated pods' logs will be unavailable using this command.

As mentioned in other answers, the best way is to have your logs centralized via logging agents or directly pushing these logs into an external service.

Alternatively and given the logging architecture in Kubernetes, you might be able to fetch the logs directly from the log-rotate files in the node hosting the pods. However, this option might depend on the Kubernetes implementation as log files might be deleted when the pod eviction is triggered.

yyyyahir
  • 2,262
  • 1
  • 5
  • 14
13

From kubernetes docs:

Examples

# Return snapshot logs from pod nginx with only one container
kubectl logs nginx

# Return snapshot of previous terminated ruby container logs from pod web-1
kubectl logs -p -c ruby web-1

# Begin streaming the logs of the ruby container in pod web-1
kubectl logs -f -c ruby web-1

# Display only the most recent 20 lines of output in pod nginx
kubectl logs --tail=20 nginx

# Show all logs from pod nginx written in the last hour
kubectl logs --since=1h nginx

Options

 -c, --container="": Print the logs of this container
  -f, --follow[=false]: Specify if the logs should be streamed.
      --limit-bytes=0: Maximum bytes of logs to return. Defaults to no limit.
  -p, --previous[=false]: If true, print the logs for the previous instance of the container in a pod if it exists.
      --since=0: Only return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to all logs. Only one of since-time / since may be used.
      --since-time="": Only return logs after a specific date (RFC3339). Defaults to all logs. Only one of since-time / since may be used.
      --tail=-1: Lines of recent log file to display. Defaults to -1, showing all log lines.
      --timestamps[=false]: Include timestamps on each line in the log output

That is just a simple way of doing it. But in production , I would send all the logs of all the pods to a centeral Log management system such as ELK by deploying a log sending client on the kubernetes cluster as daemon-set such as fluentbit , that will keep sending logs to ELk where I am able to filter things base don the namespace , pod , container , or any other label.

Ijaz Ahmad
  • 11,198
  • 9
  • 53
  • 73
3
kubectl get event -o custom-columns=NAME:.metadata.name -n <namespace> --no-headers

use the above command to get the list of terminated pods in your namespace and use

kubectl logs -f pod-name -n <namespace> -p

to see the terminated pod's logs

P.S.: The above command to fetch terminated pod details will give you the pods which were terminated 1 hour ag

Fabrice Jammes
  • 2,275
  • 1
  • 26
  • 39
Sakthi Sri S V
  • 39
  • 1
0

You can try --previous flag on the logs

i.e.

kubectl --namespace namespace logs pod_name --previous

This will show the logs from dump pod logs (stdout) for a previous container accroding to kubernetes docs

ron
  • 726
  • 11
  • 16
0

A combination of a flag --previous and a container name for a container that was terminated with reason: CrashLoopBackOff: First find a pod in a namespace. Its status is CrashLoopBackOff:

kubectl get pods -n namespace_name

NAME READY STATUS RESTARTS AGE crashing_pod_name 0/9 Init:CrashLoopBackOff 17 (105s ago) 63m

Then use describe to find out a name of a container that failed:

kubectl describe pod -n namespace_name crashing_pod_name

Find a name of a container that is terminated with a reason CrashLoopBackOff

Then list logs:

kubectl logs -n namespace_name crashing_pod_name -c failing_container_name --previous

user18236866
  • 1
  • 1
0

Or you can do it on the CRI (container runtime interface) level (not kubernetes-api level). For instance:

  1. See all containers (also the ones that exited): crictl ps --all
  2. Get container logs: crictl logs <pod-id>

Few popular CRI for Kubernetes for more details:

CLI for them should be the same (the crictl ...)

Andreas
  • 41
  • 5