2

I am using Prometheus and have a counter measuring the number of times an API is called. Prometheus resets the counter whenever the API goes down but stores the previous data. As you can see in the image below, the API reaches 20 calls, stops, and then restarts. For some reason, Prometheus does not start counting again from 20 when the API starts back up. Does anyone know how to edit Prometheus so that if my API restarts, Prometheus will pick up counting where it left off instead of resetting the counter? (I have a persistent storage claim set up already)

I am also using Grafana. Is there a way to ignore this drop in Grafana?

Thank you. enter image description here

Semi-okay Coder
  • 39
  • 6

1 Answers1

1

Just wrap the query into increase() function with the lookbehind window in square brackets, which covers all the samples stored in Prometheus:

increase(http_requests_total[10y])

This query removes counter resets and returns non-decreasing counter value.

It isn't recommended to use too long lookbehind windows in square brackets (e.g. 10y as in the query above), since they instruct Prometheus to read and process all the raw samples on the given lookbehind window for time series matching the given series selector. If the number of selected samples is big, the query can be slow to execute.

It is better to use the following queries with much smaller lookbehind windows in square brackets instead:

  • increase(http_requests_total[1h]) - returns the number of requests during the last hour ending at the given point on the graph. Note that the query is executed independently per each point on the graph. See increase() function docs.
  • rate(http_requests_total[5m]) - returns the average per-second requests rate over the last 5 minutes. See rate() function docs.
valyala
  • 11,669
  • 1
  • 59
  • 62