prometheus scrape-interval 1d

Question

I'm working with prometheus to scrape k8s service metrics.

I created a service monitor for my service as below :

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: scmpoll-service-monitor-{{ .Release.Name }}
  namespace: {{ .Release.Namespace }}
  labels:
    app: scmpoll-{{ template "jenkins-exporter.name" . }}
    chart: {{ template "jenkins-exporter.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  namespaceSelector:
    matchNames:
    - {{ .Release.Namespace }}
  selector:
    matchLabels:
      app: scmpoll-{{ template "jenkins-exporter.name" . }}
      chart: {{ template "jenkins-exporter.chart" . }}
      release: {{ .Release.Name }}
  endpoints:
  - interval: 1440m
    targetPort: 9759
    path: /metrics
    port: http

I set interval: 1440m because I want prometheus to scrape data once a day.

After deploying the chart, the service monitor was added to prometheus targets but with status unknown and scrape duration 0s. The screenshot below:

Also I have the service monitor added to prometheus config:

- job_name: monitoring/scmpoll-service-monitor-my-release/0
  honor_timestamps: true
  scrape_interval: 1d
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http

I can see that scrape-intervel is set to 1d (24h) but the state is unknown in the screenshot above. Do I have to wait for the next 24h and check or does this mean the configuration is wrong?

Test : i made test with scrape 20minutes and it was the same thing : status unknow with no metrics until the 20 minutes passed and status became UP and metrics were scraped.

I'm not working on prometheus-operator chart, it's an independent chart.

k8s 1.16 and prometheus stack 12.11;; i am already using pushgateway! at least isn't there any solution to use cron expression for scrapping? — ben amara marwa, Nov 29 '21 at 09:02
Do you use any particular labels in ServiceMonitor selector in your Prometheus setup?(serviceMonitorSelector) — Mikolaj S., Nov 29 '21 at 10:39

score 1 · Answer 1 · answered Nov 26 '21 at 13:02

1

It's not recommended to use Prometheus for such long scrape_interval. 2 minutes is suggested by many. Read this for details- Staleness.

If you want to scrape data with interval exceeding 2 minutes, you can use VictoriaMetrics. It supports time series with arbitrary long scrape intervals.

answered Nov 26 '21 at 13:02

Alif Biswas

348
1
7

i am already using pushgateway! at least isn't there any solution to use cron expression for scrapping? – ben amara marwa Nov 29 '21 at 08:58

score 1 · Answer 2 · answered Nov 26 '21 at 20:55

As mentioned, in general it's not advisable to use scrape intervals of more than 2 minutes in Prometheus (e.g. see here). This is due to the default staleness period of 5 minutes, which means that a scrape interval of 2 minutes allows for one failed scrape without the metrics being treated as stale.

There's nothing wrong with scraping a target more often. So, you can leave the scrape interval at e.g. 60 seconds, even if the metrics don't change often.

If you can't scrape the target so often for some reason, you can use a Pushgateway. The target pushes its metrics to the Pushgateway at its own pace (e.g. once per day) and Prometheus scrapes the metrics from the Pushgateway in its own interval (e.g. every 60 seconds).

i am already using pushgateway! at least isn't there any solution to use cron expression for scrapping? — ben amara marwa, Nov 29 '21 at 08:57
No, there isn't. In Prometheus you can only set the scrape interval directly, and, as mentioned, it's not intended to be longer than a few minutes. Here's also an article discussing more about the scrape interval https://www.robustperception.io/keep-it-simple-scrape_interval-id — weibeld, Nov 29 '21 at 13:04

prometheus scrape-interval 1d

2 Answers2