1

Is it possible to aggregate 2 gauge metrics (i.e. kube_pod_labels and kube_pod_container_resource_requests_cpu_cores) so that by executing the query both resulting elements would be combined (i.e. all pod labels as resulting element and request_cpu_cores as value)?

Query for cpu request value looks like this sum by (namespace, pod) (kube_pod_container_resource_requests_cpu_cores{cluster="my-cluster"})

Actual result:

{namespace="my-ns",pod="my-pod"} 0.05

Query for pod labels is kube_pod_labels{label_foo="bar"}

Actual result:

kube_pod_labels{cluster="my-cluster",label_foo="bar",label_app="my-app-label",namespace="my-ns",pod="my-pod",service="my-svc"} 1

I have tried using a left joint but it seems that grouping by a given label (pod, namespace etc) is required as explained in this https://www.robustperception.io/left-joins-in-promql.

With a multiplication operator * is possible to obtain the desired result set but the set would only contain labels specified in the by clause. Example query:

group by (namespace,pod) (kube_pod_labels{label_foo="bar",cluster="my-cluster"}) * sum by (namespace, pod) (kube_pod_container_resource_requests_cpu_cores{cluster="my-cluster"})

Example result:

{namespace="my-ns",pod="my-pod"}    0.05

What I am trying to obtain is a resulting set containing all labels without having to filter by an arbitrary label/value

The desired result by joining the 2 queries should be:

{cluster="my-cluster",label_foo="bar", label_app="my-app-label",namespace="my-ns",pod="my-pod",service="my-svc"} 0.05
srodi
  • 41
  • 10
  • 1
    hello and welcome to SO. Yes it is possible, please take a look at this very similar question/answer https://stackoverflow.com/a/68679095/2096986 and in case you have a different use case please specify in your question. – Felipe Aug 11 '21 at 13:40
  • Thanks Felipe but not sure if that is helpful for my use case. I have update the question with an example of actual results for the 2 queries I currently have and the expected result for the "joint_query" I need help defining. I hope that explains a little better what I am trying to achieve. – srodi Aug 11 '21 at 14:19
  • what you explained in your question is most likely what I explain in my other answer. `metric1` is your `kube_pod_labels` and `metric2` is your `kube_pod_labels`. Just join then with `* on (id) group_left()` in the middle and replace `id` by `cluster` from your labels. Then I suppose you will have no result because the labels of both metrics don't match. And then you need to use `label_replace`. I don't have you prometheus dashboard and metrics available. So, please give a educated attempt and improve your question so I can try to answer. – Felipe Aug 11 '21 at 15:16
  • Felipe, could you please clarify what should be `metric1` and `metric2`? In your last answer you mentioned `kube_pod_labels` for both.. also, please see my latest question edit where I am filtering the results of `kube_pod_labels` based on `label_foo="bar"`. thank you – srodi Aug 11 '21 at 15:52
  • sorry, `metric1` is your `sum by (namespace, pod) (kube_pod_container_resource_requests_cpu_cores{cluster="my-cluster"})` and `metric2` is your `kube_pod_labels{label_foo="bar"}`. join both using `* on (cluster) group_left()`. I guess it will be something like: `sum by (namespace, pod) (kube_pod_container_resource_requests_cpu_cores{cluster="my-cluster"}) * on (cluster) group_left() kube_pod_labels{label_foo="bar"}`. maybe you have to exclude some of the labels using `without(....)` – Felipe Aug 11 '21 at 17:50

1 Answers1

3

This can be achieved with a combination of the following:

  • label_replace query function: For each timeseries in v, label_replace(v instant-vector, dst_label string, replacement string, src_label string, regex string) matches the regular expression regex against the value of the label src_label. If it matches, the value of the label dst_label in the returned timeseries will be the expansion of replacement, together with the original labels in the input. Capturing groups in the regular expression can be referenced with $1, $2, etc. If the regular expression doesn't match then the timeseries is returned unchanged. https://prometheus.io/docs/prometheus/latest/querying/functions/#label_replace

  • multiplication * operator and group_left() modifier: Many-to-one and one-to-many matchings refer to the case where each vector element on the "one"-side can match with multiple elements on the "many"-side. This has to be explicitly requested using the group_left or group_right modifier, where left/right determines which vector has the higher cardinality. https://prometheus.io/docs/prometheus/latest/querying/operators/

Example query:

label_replace(kube_pod_labels{},"label","$1","label_", "(.+)") 
* on (cluster,namespace, pod) group_left() 
(sum by (cluster,namespace, pod) (kube_pod_container_resource_requests_cpu_cores{}))

Note that: If the regular expression doesn't match then the timeseries is returned unchanged. In this case the regular expression does not match - hence the full set of labels is return unchanged.

Example result:

{cluster="my-cluster",label_foo="bar", label_app="my-app-label",namespace="my-ns",pod="my-pod",service="my-svc"} 0.05

Felipe provided a valuable hint on how to achieve this result in a comment for the original question.

srodi
  • 41
  • 10