Graph a counter from zero in prometheus/grafana

Question

In prometheus, I have a monotonically increasing counter (ifHCInOctets from IF-MIB, in this case).

In Grafana, I can create a graph using the simple query ifHCInOctets{job='snmp',instance='$Device',ifDescr=~'eth0'} and see the counter graphed over different time ranges by selecting the desired range in the upper-right.

This is almost exactly what I want. However, I would like the graph to always start at zero and increase from there. The use-case is that I want to visualize my data usage over the course of a month to see how quickly I am approaching my data cap. (I already create a gauge object using increase(ifHCInOctets{...}[$__range]) function which shows me how much I have used in total over the given time range, but I'd like to be able to visualize that usage over time.)

Basically, I want ifHCInOctets{...} - X where X is the value of ifHCInOctets at the start of the range. My first thought was:

ifHCInOctets{...} - ifHCInOctets{...} offset $__range

But that seems to show me each data point minus the data point $__range time prior to it (rather than just subtracting the starting value from all points).

I then tried creating a query variable with the query query_result(ifHCInOctets{...} offset $__range) and setting it to update on time range change. This almost seemed to work, but the resulting graph always seemed to start slightly negative, depending on the time range selected, which made me think it wasn't doing what I thought it was.

I have also tried various forms of sum, sum_over_time, and increase, all to no avail.

Answer 1

You're probably looking for something like this

ifHCInOctets
  -
min_over_time(
  (ifHCInOctets
    and
  (month(timestamp(ifHCInOctets)) == scalar(month(vector($__to / 1000)))))[31d:]
)

But it doesn't take into account counter resets. And is ugly and inefficient as hell. It's basically the current value minus the min_over_time calculated over samples in the previous 31 days that fell into the same month as Grafana's $__to timestamp.

You probably want to set up a recording rule based on this expression (that adds year, month and day labels to a metric) and then calculate the increase() over any given month (including the current month). That takes into account both counter resets and counters that did not exist at the beginning of the month.