5

I just want to represent the CPU metrics of a POD as below

enter image description here

I am able to represent the CPU requests and limits in terms of CPU cores which are directly available through prometheus scrape metrics.

But in prometheus I don't see any direct metric to get the CPU cores used by a POD so, Can someone give me a work around or a way to represent the CPU usage of a POD in the terms of CPU cores.

Thanks in advance

Ramana
  • 63
  • 1
  • 6

2 Answers2

7

The query you are looking for is this one: 

sum(rate(container_cpu_usage_seconds_total{container_name!="POD"}[1m])) by (pod_name)

Here the explanation (from within to out, as Prometheus calculates this query):

  • container_cpu_usage_seconds_total which gives you how long the CPU has been occupied. 1s = a single Core for a whole second
  • {container_name!="POD"} ignores the meta cGroups.
  • rate(....[1m]) gives you the value changes in the defined interval, here 1 minute
  • sum(....) by (pod_name) add up all values which contain the same pod names, which will be the case if we have multiple containers in the same pod

For further information about Prometheus in Kubernetes you can read this blog here: https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-part-3-container-resource-metrics-361c5ee46e66

secustor
  • 3,001
  • 2
  • 14
  • 20
  • thanks for the info but when I apply above principle I observed the following things as shown in the below graph [pod cpu uage]: https://i.stack.imgur.com/vV16Q.jpg though the cpu usage is not at near the cpu limit (even after I multiply the resultant value by 100) the cpu started throttling. So I think the above principle you have mentioned is not accurate or please help me understand If I am doing anything wrong – Ramana May 13 '20 at 08:22
  • by the way below is my pod resource config resources: requests: memory: "64Mi" cpu: "250m" limits: memory: "128Mi" cpu: "500m" – Ramana May 13 '20 at 08:40
  • the above Prometheus query works fine, but somehow in Minikube, the pod will start throttling even though the CPU usage is not near the defined limits. – Ramana Sep 13 '20 at 07:07
4

The container_name and pod_name labels have been renamed to container and pod starting from Kuberntes 1.16 - see this issue for details. So for Kubernetes 1.16+ the following PromQL query must be used for calculating per-pod CPU usage in CPU cores:

sum(rate(container_cpu_usage_seconds_total{container!~"POD|"}[5m])) by (pod)
valyala
  • 11,669
  • 1
  • 59
  • 62