Get Total requests in a period of time

Question

I need to show, in Grafana, a panel with the number of requests in the period of time selected in the upper right corner.

For this I need to solve 2 issues here, I will ask the prometheus question here and the Grafana question in another link.

If I have a Counter http_requests_total, How can I build a query to get an integer with the total number of requests during a period of time (for example:24hs)?

Note that Prometheus may return fractional value from `increase()` function on a time series with integer values. It may also miss some values on slowly increasing time series. Both issues are documented at https://github.com/prometheus/prometheus/issues/3746 . If you need accurate integer values from `increase()` function, then take a look at [MetricsQL](https://victoriametrics.github.io/MetricsQL.html). — valyala, Dec 05 '20 at 21:55

score 248 · Accepted Answer · answered Nov 08 '17 at 07:51

248

What you need is the increase() function, that will calculate the difference between the counter values at the start and at the end of the specified time interval. It also correctly handles counter resets during that time period (if any).

increase(http_requests_total[24h])

If you have multiple counters http_requests_total (e.g. from multiple instances) and you need to get the cumulative count of requests, use the sum() operator:

sum(increase(http_requests_total[24h]))

See also my answer to that part of the question about using Grafana's time range selection in queries.

answered Nov 08 '17 at 07:51

Yoory N.

4,881
4
23
28

9

What if period is not last 24h, but from fist date&time and second date&time? – Cherry Mar 26 '19 at 10:00
@Cherry, You can use an offset, e.g. `increase(http_requests_total[5h] offset 1d)` or `increase(http_requests_total[357s] offset 123m)`. This way you specify the width of the period you interested in and how far in the past this period is. But this is definitely NOT a convenient way. Maybe somebody else can suggest a more practical solution to your question. [@donotreply's answer](https://stackoverflow.com/a/54776593/8868289) looks like what you ask for, but seems to be applicable only when using Graphana. – Yoory N. Mar 27 '19 at 06:11
5

Actually on server restart prometheus values get reset to 0, so the graph suddenly drops, if we see the increase of 24 hr, it comes inaccurate as it is the difference of the first and last value , any better approach to this ? – somya bhargava Apr 01 '20 at 02:13
3

@somyabhargava I had the exact problem - I found the answer on https://stackoverflow.com/questions/55928079/grafana-single-stat-after-big-counter-reset . So in this case it would be `sum(increase(http_requests_total[100y]))` – willowherb Apr 10 '20 at 15:16
2

But `sum(increase(http_requests_total[100y]))` will get you the total value over the whole lifetime of the counter and not just the selected time interval – trallnag Apr 17 '20 at 10:29
Yes exactly what @JellyFilledNuts said, this will be a whole lifetime counter – somya bhargava Apr 18 '20 at 01:52
What about count of bucket elements? If I run job_timer_myjob_bucket{le="+Inf"} it returns number of items in the bucket e.g. 100. Now I would like to get the same count for specified interval e.g. for the previous day. If I use increase(job_timer_myjob_bucket{le="+Inf"}[1h]) it return me not integer - e.g. 55.8 is returned. – StanislavL Jul 29 '21 at 14:36

score 63 · Answer 2 · answered Feb 19 '19 at 23:38

63

SO won't let me comment on Yoory's answer so I have to make a new one...

In Grafana 5.3, they introduced $__range for Prometheus that's easier to use:

sum(rate(http_requests_total[$__range]))

This variable represents the range for the current dashboard. It is calculated by to - from

http://docs.grafana.org/features/datasources/prometheus/

answered Feb 19 '19 at 23:38

donotreply

739
5
6

4

note: for me, $__range does work, but grafana does not show it as an available variable in its helper tooltip/popup. go hardcore and type it out. – Agoston Horvath Mar 04 '22 at 16:21

score 26 · Answer 3 · edited Oct 11 '19 at 12:26

26

As per increase() documentation, it is not aggregation operator. Thus, it will give wrong answer. (See note.)

You should use sum_over_time() function which aggregates over time interval.

sum_over_time(http_requests_total[24h])

If you have multiple counters, use sum() operator:

sum(sum_over_time(http_requests_total[24h]))

Note: I have 5 datapoints which has values: 847, 870, 836, 802, 836. (updated every minute)

increase(http_requests_total[5m]) returns 2118.75 

sum_over_time(http_requests_total[5m]) returns 4191

edited Oct 11 '19 at 12:26

Optimus Prime

6,817
5
32
60

answered Sep 26 '19 at 15:06

foxtrot9

491
4
9

6

sum_over_time shouldn't be used directly on counters, use after rate. – Optimus Prime Mar 11 '20 at 10:26
2

When I use `sum(sum_over_time(http_requests_total[$__interval]))` I am still seeing drops in the graph while instead it should be monotonously increasing. – tcurdt May 04 '21 at 22:55
1

The data points are not monotonically increasing. Is the counter resetting every minute and you're collecting every minute? Are you performing a push method of getting data into prometheus? This would explain why sum_over_time works for you. For others who's data points would be scraped a reset cannot happen every minute, so you would find counters that look like this: 847, 1690, 2412, 3245, 4023. In this case the increase function should work better. – Michael C. Chen Oct 09 '21 at 01:38

score 18 · Answer 4 · answered Sep 29 '20 at 16:51

http_requests_total - http_requests_total offset $__interval > 0

This builds off another answer and comment that works and handles restart situations.

The offset keeps the value always as an integer and does not try to perform interpolation like the increase and rate functions.

The > 0 filter at the end will ignore all of the negative values that could be captured due to a restart.

The end result is the accurate total number of requests over time if you choose to chose the total value in the legend.

All the answers using increase() didn't work across all dashboard ranges, this one does! — Markus, Nov 05 '21 at 13:29

score 12 · Answer 5 · answered May 13 '20 at 09:46

Solution: In order to calculate sum of https counters on prometheus grafana you should use increase method and set generic Time Range $interval in order to sum and calculate all http requests counters.

increase(http_requests_total[$interval])

According to Prometheus Reference:

increase() increase(v range-vector) calculates the increase in the time series in the range vector. Breaks in monotonicity (such as counter resets due to target restarts) are automatically adjusted for. The increase is extrapolated to cover the full time range as specified in the range vector selector, so that it is possible to get a non-integer result even if a counter increases only by integer increments.

The following example expression returns the number of HTTP requests as measured over the last 5 minutes, per time series in the range vector:

increase(http_requests_total{job="api-server"}[5m]) increase should only be used with counters. It is syntactic sugar for rate(v) multiplied by the number of seconds under the specified time range window, and should be used primarily for human readability. Use rate in recording rules so that increases are tracked consistently on a per-second basis.

P.S

You should set the correct Quick range on Grafana for setting the right time frame you choose (that straight rendered to $interval variable) In addition I suggest to set on the Graph visualisation the right resolution and Min time interval ( in your case it's per day -> 1d)

2.In order to sum all amount of requests just perform sum function

sum(increase(http_requests_total[$interval]))

I used `sum(increase(http_requests_total[$interval]))` query and visualized it in Grafana bar chart. I would expect that the far right bar would display today's data. However, it displays yesterday's data. Moreover, the date below the bar is incorrect, it is one day in the future (it displays `7-Apr-2022` although yesterday was `6-Apr-2022`). — Sam Carlson, Apr 07 '22 at 06:20

score 4 · Answer 6 · answered Jan 15 '20 at 23:03

4

To get the exact count for the last 24 for hours I have created the following query:

max_over_time(http_requests_total[6s])- min_over_time(http_requests_total[24h])

Note: works for me :)

answered Jan 15 '20 at 23:03

Andrii Soluk

520
1
6
14

21

This will give you wrong results if `http_requests_total` has been reset during those 24 hours. – trallnag Apr 22 '20 at 21:37

score 3 · Answer 7 · answered Dec 19 '22 at 08:27

3

The easiest way to do this in the modern version of Grafana is to set the Calculation field of the Value options section to "Difference" reduce function:

The query is your http_requests_total data without any aggregation.

answered Dec 19 '22 at 08:27

Vladislav

1,696
27
37

thx, this is the most accurate answers – rsobies Feb 21 '23 at 06:25
1

..but this aproach won't be restart proof, that's the disadvantage – rsobies Feb 21 '23 at 11:22

score 1 · Answer 8 · answered Feb 14 '22 at 16:05

1

It seems to me that all the previous answers have misinterpreted the questions, which is to get a count from t0 to t1, where the value at t0 should be 0.

For this one can use the @ modifier as per the documentation https://prometheus.io/docs/prometheus/latest/querying/basics/#modifier:

http_requests_total - http_requests_total @ start()

answered Feb 14 '22 at 16:05

glep

104
9

I think this one will not give the correct result in case of counter reset. – Duy Tran Aug 31 '23 at 06:11

Snaper · Answer 9 · 2022-06-29T10:55:10.483

Recently, I have the confusion too, and I got some solutions for it, but all of them are not all work perfectly.

solution 1:

    sum(increase(your_point))[$__interval]

This one will cause some different value with same statement, and also will cause the zero-value (actually not zero).

Solution 2:

    max_over_time(your_point[$__range])- min_over_time(your_point[$__range])

    your_point[$__range] -  your_point offset $__range

Both of this have potentially bug (value reset), and only can get the value in [sometime-now], cannot get the answer in anytime period.

Solution 3:

    sum_over_time(your_point)[$__range]

This solution will take much time to change your metrics (reset in certain period), but truely work.

Is anyone can give me another solution?

Solution 3 doesn't work for me: " parse error: ranges only allowed for vector selectors: details:" — Alexey Sh., Feb 13 '23 at 13:01

score 0 · Answer 10 · answered Jun 18 '20 at 14:30

0

To get the accurate total requests in a period of time, we can use offset:

http_requests_total - http_requests_total offset 24h

increase will extrapolate the range so that we can see float number in the result.

By using offset, the value is always integer because it just calculates the difference between start and end

answered Jun 18 '20 at 14:30

Haoyuan Ge

3,379
3
24
40

1

Thanks a lot! This is a most accurate request to gain real results from *_total countes. A bit more universal approach for fine-grained intervals is "http_requests_total - http_requests_total offset $__interval" (for grafana) – uptoyou Jul 14 '20 at 16:09
15

No, this is not the correct solution. If the instances are restarted, the counter will be reset. So .... – sanigo Jul 20 '20 at 07:06

Get Total requests in a period of time

10 Answers10

Linked