15

I used Prometheus to measure business metrics like:

# HELP items_waiting_total Total number of items in a queue
# TYPE items_waiting_total gauge
items_waiting_total 149

I would like to keep this data for very long term (5 years retention) and I don't need high frequency in scrape_interval. So I set up scrape_interval: "900s".

When I check the graph in Prometheus with 60s resolution, it shows that flapping, but it is not true.

enter image description here

The question is, what is the maximum (recommended) scrape_interval in Prometheus?

oliver nadj
  • 788
  • 7
  • 21
  • 1
    Prometheus explicitly isn't for long term storage of metrics. They recommend exporting to influxdb or something similar to support the storage use case. – Ben Mathews Jan 11 '18 at 21:04
  • 1
    If you need storing data with scrape interval exceeding 2 minutes while maintaining compatibility with PromQL, then try storing the data to [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/). It supports time series with arbitrary long scrape intervals. – valyala Dec 05 '20 at 22:41

3 Answers3

19

It's not advisable to go above about 2 minutes. This is as staleness is 5 minutes by default (which is what's causing the gaps), and you want to allow for a failed scrape.

brian-brazil
  • 31,678
  • 6
  • 93
  • 86
  • Thanks @brian-brazil I'll set scrape_interval to 2 minutes. – oliver nadj Oct 25 '16 at 08:46
  • Seems to be documented in [Staleness](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness) and can be configured with `--query.lookback-delta` command line flag. – dastrobu Mar 03 '21 at 05:36
2

If you want to ignore gaps, it is possible to use some aggregation_over_time functions to get your DATA from Prometheus.

max_over_time(items_waiting_total[900s])

This is useful for situations where frequent gathering of DATA is expensive for collector.

Sasha Golikov
  • 638
  • 5
  • 11
0

By default Prometheus fills gaps for up to 5 minutes after each raw sample stored in the database. See these docs for details. If you need to fill bigger gaps between raw samples, then the last_over_time function can help. Just specify the maximum gap duration in square brackets in order to fill gaps. For example, the following query would fill gaps for items_waiting_total time series for up to 900 seconds:

last_over_time(itmes_waiting_total[900s])
valyala
  • 11,669
  • 1
  • 59
  • 62