8

I have read through a bunch of guides showing how to monitor cpu and memory usage of pods in Kubernetes with Prometheus and most of them look something like this:

rate(container_cpu_usage_seconds_total{pod=~"compute-.*", image!="", container!="POD"}[5m])

but I can't find any documentation on why the container label is there (it seems like it causes duplicated data) and why it is being avoided in many monitoring examples. I know that this metric comes from the cadvisor component of Kubernetes, but the only docs I can find on these metrics are the short descriptions provided in the code here.

Does anyone know what this label is for and where there are more in depth documentation no these metrics?

nicktorba
  • 125
  • 1
  • 5
  • Could you please provide additional information about guides, where you found this information? - links, screenshots,... It would be helpful to better understand the problem. – Andrew Skorkin Aug 09 '21 at 15:20
  • It is the "pause container" described [here](https://stackoverflow.com/a/48654926/423112). – Ali Sattari Aug 11 '21 at 08:38

1 Answers1

14

Containers, as @Ali Sattari already mentioned right in the comment, are pause containers.

Pause containers

Pause container starts first, before scheduling other pods. Purpose of the pause container (container_name="POD") is to provide the network namespace for the pod and additional containers, which will be assigned to that pod. Image of the pause container is always present in Kubernetes. Due to this, allocation of the pod’s resources is instantaneous. After the pause container started, there is no other work for him.

By default, pause containers are hidden, but you can see them by running next command: docker ps | grep pause

$ docker ps | grep pause
3bb5065dd9ba        k8s.gcr.io/pause:3.2                      "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_kubernetes-bootcamp-fb5c67579-5rxjn_default_93ce94f8-b440-4b4f-9e4e-25f97be8196f_0
0627138518e1        k8s.gcr.io/pause:3.2                      "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_metrics-server-56c4f8c9d6-vf2zg_kube-system_93626697-8cd0-4fff-86d3-245c23d74a42_0
81ca597ed3ff        k8s.gcr.io/pause:3.2                      "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_storage-provisioner_kube-system_dbdec6e5-d3ed-4967-a042-1747f8bdc39a_0
0d01130b158f        k8s.gcr.io/pause:3.2                      "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_kubernetes-dashboard-968bcb79-pxmzb_kubernetes-dashboard_b1265ad7-2bce-46aa-8764-d06d72856633_0
d8a159b6215e        k8s.gcr.io/pause:3.2                      "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_dashboard-metrics-scraper-f6647bd8c-hqm6k_kubernetes-dashboard_bde40acc-a8ca-451a-9868-26e86ccafecb_0
294e81edf0be        k8s.gcr.io/pause:3.2                      "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_coredns-74ff55c5b-84vr7_kube-system_28275e83-613a-4a09-8ace-13d6e831c1bf_0
2b3bfad1201b        k8s.gcr.io/pause:3.2                      "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_kube-proxy-zxjgc_kube-system_34f8158a-487e-4d00-80f1-37b67b72865e_0
d5542091730b        k8s.gcr.io/pause:3.2                      "/pause"                 4 minutes ago       Up 4 minutes                            k8s_POD_kube-scheduler-minikube_kube-system_6b4a0ee8b3d15a1c2e47c15d32e6eb0d_0
b87163ed2c0a        k8s.gcr.io/pause:3.2                      "/pause"                 4 minutes ago       Up 4 minutes                            k8s_POD_kube-controller-manager-minikube_kube-system_57b8c22dbe6410e4bd36cf14b0f8bdc7_0
c97ed96ded60        k8s.gcr.io/pause:3.2                      "/pause"                 4 minutes ago       Up 4 minutes                            k8s_POD_etcd-minikube_kube-system_62a7db7bebf35458f2365f79293db6d3_0
4ab2d11317ed        k8s.gcr.io/pause:3.2                      "/pause"                 4 minutes ago       Up 4 minutes                            k8s_POD_kube-apiserver-minikube_kube-system_dc477bf6fc026f57469b47d9be68a88c_0

You can read more about pause containers here.

Pause containers in Prometheus

In examples provided for Prometheus you can often see the next limitation: container_name!="POD", since it's useful to request resource usage just for necessary containers that currently work, without information for pause containers.

Andrew Skorkin
  • 1,147
  • 3
  • 11