How to monitor Kubernetes Pods falling over?

Question

I'm using Kubernetes in a Google Container Engine cluster and I can see in the Stackdriver logs that one of the Pods is falling over and automatically restarting because of an OutOfMemory exception.

Is there any standard way of monitoring these events?

I'm using Heapster and it doesn't seem to provide any metrics about Pods shutting down and restarting.

Fabien · Answer 1 · 2017-06-23T12:19:09.310

3

According to our IT Sysadmin, most of the current solutions for real-time monitoring and alerting on pods failures are currently unstable or very lackluster.

We ended up developing a small script which uses the Slack.com webservice for mail notifications and such.

I am sorry if my answer is not a ready-to-use one click solution :-)

This is a real feedback from our current experience and searches.

I expect things will move fast in the near future on that topic!

Our code (simple solution for effective results):

https://github.com/OpenSensee/K8SWatch

edited Jun 23 '17 at 12:19

answered Jun 23 '17 at 09:50

Fabien

4,862
2
19
33

thanks for your feedback, we may need to build our own solution too then. Is the script you built open source? – cahen Jun 23 '17 at 10:02
1

We published our code just for you! Edited in my answer. – Fabien Jun 23 '17 at 12:18
awesome! Thanks a lot :D – cahen Jun 23 '17 at 13:02

score 2 · Accepted Answer · answered Jun 24 '17 at 12:21

There is a tool called kube-state-metrics (1), which provides metrics about k8s objects, including the restart count of containers (2). These metrics can be used by Prometheus (3), where you could create an alert if the restart count is higher than a specified number.

The Prometheus Operator of CoreOS (4) and their example configuration (5) might be useful if you decide to go this way, it was very helpful when we recently deployed it to our cluster. There is no predefined alert for the restart count in that example, but it should be easy to add one.

(1) https://github.com/kubernetes/kube-state-metrics
(2) https://github.com/kubernetes/kube-state-metrics/blob/master/Documentation/pod-metrics.md
(3) https://prometheus.io/
(4) https://coreos.com/operators/prometheus/docs/latest/user-guides/getting-started.html
(5) https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus

score 0 · Answer 3 · edited Jan 25 '22 at 04:40

0

You can also use kubernetes event exporter to catch the critical events of pods failure like BackOff, Failed, Failed Scheduling, OOMKilled etc. It is easy to setup. Here's a Setup Guide on my blog.

edited Jan 25 '22 at 04:40

cigien

57,834
11
73
112

answered Jan 22 '22 at 18:58

shawon

984
7
15

When linking to your own site or content (or content that you are affiliated with), you [must disclose your affiliation _in the answer_](/help/promotion) in order for it not to be considered spam. Having the same text in your username as the URL or mentioning it in your profile is not considered sufficient disclosure under Stack Exchange policy. – cigien Jan 23 '22 at 04:25

How to monitor Kubernetes Pods falling over?

3 Answers3