9

I am trying to create a system health check. I would like to be able to determine how often the pods are restarting. Since the pods have liveness probes, they may continuously restart if something is wrong. Is there anyway to check the time a pod restarted at instead of just the number of restarts and the total time the pod has been alive?

beanwa
  • 222
  • 2
  • 4
  • 11
  • Do you already have Prometheus (or a similar metrics gathering system) running against your cluster? – mdaniel Aug 21 '18 at 02:00
  • 1
    Separately, while not _exactly_ the answer to your question, `kubectl get --all-namespaces=true events --watch` will create a running list of all Pod events in your system, which for sure will include Pod creation and destruction. I would fully expect you could subscribe to that list just like `kubectl` does – mdaniel Aug 21 '18 at 02:01
  • @Matthew Thank you for the response. I do not believe we have prometheus (if we do, we are not utilizing it) I ran 'find / -iname '*prometheus*'' and did not find any prometheus files. I currently do not have my deployment available, will the command you mentioned give all events for all pods (will it show that pod A Started at time x failed at time y and started again at time z, or just the events of the current living pods) – beanwa Aug 21 '18 at 15:25
  • *will the command you mentioned give all events for all pods* it will, yes, that's what the `--watch` does: inform k8s that you wish to be informed of future changes to that resource, also – mdaniel Aug 22 '18 at 00:23

1 Answers1

4

You may want to use a monitoring tool, like cAdvisor/Heapster/Grafana/Prometheus.

Another option you have is to deploy the kube-state-metrics container.

Good articles to read: Monitoring Kubernetes and Get Kubernetes Cluster Metrics with Prometheus. Also, read a similar question on stackoverflow.

Vit
  • 7,740
  • 15
  • 40