2

I am monitoring some metrics for AWS SES. In periods where no mails are sent there are no datapoints present (showing as gaps on the graph). To fill out the gaps, I am using the following:

aws_ses_send_sum or on() vector(0)

and I am receving the graph visible in the screenshot.

What I want to achieve now: I would like the values with '0' to be also labeled with aws_ses_send_sum{exported_job="aws_ses",instance="localhost:9000",job="aws"} instead of appearing as: {}.

I have checked several other questions:
Merge/join two metrics in Prometheus/PromQL

How to merge zero values (vector(0) with metric values in PromQL

How to execute multiple queries in one call in Prometheus

checked the querying section in the docs but I am unable to get it right.

sum() is delivering the graph that I need - but as a result the labels are being removed.

Could someone please help?

enter image description here

enter image description here

szark
  • 77
  • 10

2 Answers2

0

I came across a similar problem and couldn't find an "official" solution either. I ended up creating a recording rule with my expression and used that as a workaround. I don't know if it would help with the labels but it solves the metric name problem at least.

ETA some more details on the rule config and info on preserving labels.

First, we need a zero-vector that has the appropriate labels for us to sum by later. Based on this post, we can use clamp_max(absent(notExists{exported_job="aws_ses",...}),0) (... marks the other labels and values you listed, such as instance).

Next, we combine the two time series with a simple or: aws_ses_send_sum{...} or clamp_max(...)

Finally, we can sum by each label to preserve them all in the resulting time series: sum(... or ...) by (exported_job="aws_ses",...)

Now we have a single time series with the appropriate labels but without a name. This is where rules come into play: take this whole expression that we just created, and add an entry in your rules.yaml file.

groups:
  - name: label_name_preserver
    rules:
    - record: aws_ses_send_sum_zero
      expr: sum(...) by(...)

Please note that I have no idea what a metric with a same name as a rule could cause (if anything), so it would be best to assign some other name (e.g. aws_ses_send_sum_zero).

nfanni97
  • 1
  • 1
  • could you please share a sample snipped of the recording rule. that will help improve thee quality of your answer. thanks – Ouss Dec 29 '21 at 08:52
  • 1
    Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Dec 29 '21 at 08:52
0

I'm unaware of easy solution for PromQL, but the task can be easily solved with default operator in MetricsQL:

aws_ses_send_sum default 0

The default operator fills gaps in the selected time series with the provided value (zero in this case). See also interpolate and keep_last_value functions in MetricsQL.

Another solution is to change the corresponding Grafana graph settings, so it automatically fills gaps (aka null values) with zero values. See Time series from Prometheus source: how to set nulls as zero?

valyala
  • 11,669
  • 1
  • 59
  • 62