13

I need to get CPU and Memory usage in kubernetes pods with prometheus queries. Can someone plz help?

GihanS
  • 445
  • 2
  • 5
  • 15
  • 1
    Please provide more information on your current situation. Is Prometheus up and running but you don't know how to query for metrics? Are you having trouble getting Prometheus running in your cluster? – Monkeyanator Mar 13 '19 at 14:55
  • Possible duplicate of [count k8s cluster cpu/memory usage with prometheus](https://stackoverflow.com/questions/54866777/count-k8s-cluster-cpu-memory-usage-with-prometheus) – cookiedough Mar 13 '19 at 15:16
  • yes. it's up and running I want get alerts for CPU and Memory usage of the pods. For that I need to have prometheus queries. Plz can I have what u r using ? – GihanS Mar 13 '19 at 15:56
  • I want to have something like this "sum(container_memory_usage_bytes{namespace="$namespace", pod_name="$pod", container_name!="POD"}) by (container_name)" Since there are variables in this query Im unable to send alerts. – GihanS Mar 13 '19 at 16:00
  • Please edit your question with whatever query you tried. – Dev Dec 02 '20 at 05:09

4 Answers4

14

For CPU percentage

avg((sum (rate (container_cpu_usage_seconds_total {container_name!="" ,pod="<Pod name>" } [5m])) by (namespace , pod, container ) / on (container , pod , namespace) ((kube_pod_container_resource_limits_cpu_cores >0)*300))*100)

For Memory percentage

avg((avg (container_memory_working_set_bytes{pod="<pod name>"}) by (container_name , pod ))/ on (container_name , pod)(avg (container_spec_memory_limit_bytes>0 ) by (container_name, pod))*100)

you can use above promql with pod name in a query.

iknow
  • 8,358
  • 12
  • 41
  • 68
Peter
  • 405
  • 3
  • 6
  • 14
4

The following query should return per-pod number of used CPU cores:

sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name!=""}[5m])) without (container_name)

The following query should return per-pod RSS memory usage:

sum(container_memory_working_set_bytes{container_name!="POD",pod_name!=""}) without (container_name)

If you need summary CPU and memory usage across all the pods in Kubernetes cluster, then just remove without (container_name) suffix from queries above.

valyala
  • 11,669
  • 1
  • 59
  • 62
0

Do you use prometheus-operator to collect data from kubernetes? If yes, you can use something like this: sum(container_memory_usage_bytes) sum(container_cpu_usage_seconds_total) Just for example.

Максим Штернберг
  • 84
  • 1
  • I want to have something like this "sum(container_memory_usage_bytes{namespace="$namespace", pod_name="$pod", container_name!="POD"}) by (container_name)" Since there are variables in this query Im unable to send alerts. – GihanS Mar 13 '19 at 16:00
0

To better monitor the memory usage of Pod/Containers. The container_memory_max_usage_bytes should also be used to monitor besides container_memory_working_set_bytes.

We used to monitor the memory usage of the pod through

avg(container_memory_working_set_bytes{container!="POD"}) by (pod)  / (avg(kube_pod_container_resource_requests_memory_bytes{container!="POD"} > 0) by (pod)) *100

However, we met the OOMKilled of the pod and failed to find anything abnormal from the above metrics.

Through the container_memory_max_usage_bytes we find the abnormal memory usage of the pod.

avg(container_memory_max_usage_bytes{ container!="POD"}) by (pod)  / (avg(kube_pod_container_resource_requests_memory_bytes{ container!="POD"} > 0) by (pod)) *100
zangw
  • 43,869
  • 19
  • 177
  • 214