0

I am using NodeJS prom-client package.

Let's say my counter increases by 1 every second but prometheus server scrapes my endpoint every 30 seconds.

Does it know what happens between the scrapes?

Or does it just see that value of the counter was 30 at t=30s and 60 at t=60s?

ablaszkiewicz1
  • 855
  • 1
  • 10
  • 26

2 Answers2

1

Prometheus knows nothing about metric values between scrapes. It stores metric values obtained only during target scrapes.

valyala
  • 11,669
  • 1
  • 59
  • 62
1

prometheus server scrapes my endpoint every 30 seconds.

does it just see that value of the counter was 30 at t=30s and 60 at t=60s?

Yes, having a scrape interval of 30 seconds DOES mean that a counter value is updated once in 30 seconds.

What does it mean in practice for your queries?

Since for a counter, you don't really care about the exact value but want to see a counter change trend over time, you are going to query for a range vector (to apply one of the rate functions). Since a rate function requires at least 2 samples, here are some hints to select a range interval:

  1. Queries with a range interval of less than the 30s (my_metric[20s]) don't make sense since you won't be able to get a range of several values

  2. A range interval of the 30s (my_metric[30s]) gives you only a single sample most of the time since a query time doesn't align with a scrape time

  3. A range interval of the 2*30s (my_metric[1m]) seems to get you reliably 2 samples in theory, but in practice, Prometheus might scrape a few milliseconds earlier or later, so due to unlucky query timing you still can get a single sample

  4. A range interval of the 3*30s (my_metric[1m30s]) seems to be robust enough, however, it's not since sometimes Prometheus might skip a scrape (let's say there is a network hiccup or whatever)

So the recommendation is that a range should be at least 4x of the scrape interval, which is 4*30s (my_metric[2m]).

star67
  • 1,505
  • 7
  • 16