5

How to add label filters in Prometheus query?

kube_pod_info

kube_pod_info{created_by_kind="ReplicaSet",created_by_name="alertmanager-6d9f74d4c5",instance="kube-state-metrics:8080",job="kube-state-metrics",namespace=“test",pod="alertmanager-6d9f74d4c5-xlqrv"}

kube_pod_labels

kube_pod_labels{instance="kube-state-metrics:8080",job="kube-state-metrics",label_app="alertmanager",label_pod_template_hash="6d9f74d4c5",namespace=“test",pod="alertmanager-6d9f74d4c5-xlqrv”,label_source=“k8s"}

Here, I have kube state metrics info in prometheus for kube_pod_info & kube_pod_labels.

kube_pod_info{namespace="test"} ---> Filters pods by namespace test.

Here, I want to include filter based on labels as well. I have a label called "label_source=“k8s" in kube_pod_labels. How can I join kube_pod_info & kube_pod_labels to apply label filter as well?

halfer
  • 19,824
  • 17
  • 99
  • 186
user1578872
  • 7,808
  • 29
  • 108
  • 206

2 Answers2

2

You can use + operator to join metrics. Here, group_left() will include the extra label: label_source from the right metric kube_pod_labels. The metric you're joining is forced to zero ( i.e. 0 * kube_pod_labels ) so that it doesn't affect the result of first metric. 

(
kube_pod_info{namespace="test"}
)
   + on(namespace) group_left(label_source)
(
   0 * kube_pod_labels
)
Kamol Hasan
  • 12,218
  • 1
  • 37
  • 46
  • Error executing query: found duplicate series for the match group {namespace="test1"} on the right hand-side of the operation: ... ..... many-to-many matching not allowed: matching labels must be unique on one side – user1578872 Feb 05 '20 at 18:09
  • @user1578872, yes, this is intended. You need to join two metrics by unique label. Like, `on(instance) group_left(label_source)`. – Kamol Hasan Feb 06 '20 at 03:25
0

As per the "Join Metrics" documentation here, you can essentially combine the labels from the kube_pod_info query with those of the kube_pod_labels query.

Something along the lines of this:

kube_pod_info{namespace="test"} * on (namespace, pod) group_left() kube_pod_labels{label_source="k8s"}

This query should give you all of the pods in the namespace test, which have the source label set to k8s.

Lawrence Warren
  • 38
  • 1
  • 8