15

Suppose I write a basic PromQL query like this

Query: kube_deployment_spec_replicas{}

Result: kube_deployment_spec_replicas{deployment="mydeployment",endpoint="myendpoint",instance="myinstance",job="myjob",namespace="default",pod="mypod",service="myservice"}

Is there a clean way to omit instance and pod from the resulted timeseries?

Desired: kube_deployment_spec_replicas{deployment="mydeployment",endpoint="myendpoint",job="myjob",namespace="default",service="myservice"}

Edward
  • 153
  • 1
  • 1
  • 6
  • Why do you want to do this? You can't just drop labels because as the result of your query you need to have unique timeseries' and simply dropping labels would violate that constraint. – Oliver Nov 19 '19 at 02:11
  • @Oliver These metrics are collected through a kube-state-metrics job. instance and pod labels change for these metrics when the job scales up and down. This causes prometheus alertmanager to potentially change the state of a prometheus alert from firing to pending. – Edward Nov 19 '19 at 18:25
  • So are all the metrics the same? Then why not go with Sergio's suggestion to use max or avg to reduce the timeseries to just one per deployment. – Oliver Nov 20 '19 at 16:02

2 Answers2

21

Try using something like the following as Sergio already suggested:

sum(kube_deployment_spec_replicas) without (instance, pod)

This query will remove instance and pod labels from the result. Note that the query won't work as expected if there are multiple kube_deployment_spec_replicas time series with identical (instance, pod) set of labels. In this case the query will sum up these time series.

p.s. MetricsQL - PromQL-like query language from Prometheus-like monitoring system - VictoriaMetrics - provides more obvious solution to delete labels with label_del function:

label_del(kube_deployment_spec_replicas, "instance", "pod")

Disclosure: I'm the author of VictoriaMetrics.

valyala
  • 11,669
  • 1
  • 59
  • 62
1

You need to use Prometheus query operators. You can use avg or sum depending on your use case.

You can check here for more information: here

Sergio Santiago
  • 1,316
  • 1
  • 11
  • 19
  • 1
    As shown in the question, my query doesn't use built in functions and using built in functions wouldn't achieve the desired behavior. Also, I'm trying to accomplish this in a clean manner (without nested label_replaces) – Edward Nov 18 '19 at 18:39