4

I have an unscheduled batch job that may or may not be triggered multiple times a day. For every invocation, I'm generating a count metric incremented by 1.

I expected to apply increase() function over 1 day for the metric and see a spike for every invocation of that day.

However, I noticed increase function doesn't show or extrapolate data for extended missing period (like the period before adding these metrics) and hence the first invocation always makes zero increments.

I'm trying something like increase(job_invocation_total[1d]).

I also tried to set the non-existent values as zero with OR on() vector(0) but increase() function didn't allow it.

Similarly for clamp_min()

I'm pretty sure there must be a standard way of achieving it but unfortunately, I can't think of it.

Captain Levi
  • 804
  • 7
  • 18

2 Answers2

1

If you have access to the code where the metric job_invocation_total is created and the cardinality of the metric labels is bound, you can initialize the counter with the value 0.

If this is not the case, you can create a recording rule that is either 0 or equal to the metric job_invocation_total.

trallnag
  • 2,041
  • 1
  • 17
  • 33
0

It's not clear what number you're asking for, but changes(push_time_seconds[1d]) would give you the number of times it pushed to the pushgateway in the past day.

brian-brazil
  • 31,678
  • 6
  • 93
  • 86
  • This may be similar to this question https://stackoverflow.com/questions/63891162/how-to-detect-a-new-metrics-with-prometheus-alerting-rule/63935356#63935356. You may want to comment on my answer since I made an assumption about increase(). – Michael Doubez Sep 17 '20 at 09:58