Stack Overflow
Stack Overflow

Questions tagged [prometheus]

The Prometheus monitoring system, including the server, alertmanager, push gateway, exporters, client libraries and other components.

Prometheus is a go-based open-source monitoring system inspired by Google's approach to monitoring.

Prometheus itself is a time-series storage server that periodically pulls metrics from monitored entities. These metrics can then be queried and alerted-upon using simple query and alert languages.

Prometheus comes with a suite of supporting tools like alertmanager, exporters and client libraries, designed to work together in providing a highly scalable and complete monitoring solution. Metric visualizations is usually done with Grafana, which seamlessly connects with Prometheus.

Prometheus's official site.

Read more about Prometheus's approach here:

  • This article explains Prometheus metric gathering approach for beginners.

  • This chapter from the online book "Site Reliability Engineering" describes Google's monitoring system Borgmon. Prometheus is mentioned in this chapter and was designed with Google's approach in mind.

6591 questions
181
votes
3 answers

Do I understand Prometheus's rate vs increase functions correctly?

I have read the Prometheus documentation carefully, but its still a bit unclear to me, so I am here to get confirmation about my understanding. (Please note that for the sake of the simplest examples possible I have used the one second for scraping…
beatrice
  • 3,684
  • 5
  • 22
  • 49
180
votes
10 answers

Get Total requests in a period of time

I need to show, in Grafana, a panel with the number of requests in the period of time selected in the upper right corner. For this I need to solve 2 issues here, I will ask the prometheus question here and the Grafana question in another link. If I…
Facundo Chambo
  • 3,088
  • 6
  • 20
  • 25
76
votes
4 answers

Usecases: InfluxDB vs. Prometheus

Following the Prometheus webpage one main difference between Prometheus and InfluxDB is the usecase: while Prometheus stores time series only InfluxDB is better geared towards storing individual events. Since there was some major work done on the…
SpaceMonkey
  • 965
  • 1
  • 8
  • 12
74
votes
2 answers

How to persist data in Prometheus running in a Docker container?

I'm developing something that needs Prometheus to persist its data between restarts. Having followed the instructions $ docker volume create a-new-volume $ docker run \ --publish 9090:9090 \ --volume a-new-volume:/prometheus-data \ …
Matt
  • 9,068
  • 12
  • 64
  • 84
61
votes
4 answers

Prometheus query to count unique label values

I want to count number of unique label values. Kind of like select count (distinct a) from hello_info For example if my metric 'hello_info' has labels a and b. I want to count number of unique a's. Here the count would be 3 for a = "1", "2", "3".…
emperorspride188
  • 819
  • 1
  • 8
  • 13
60
votes
9 answers

How do I write a Prometheus query that returns the value of a label?

I'm making a Grafana dashboard and want a panel that reports the latest version of our app. The version is reported as a label in the app_version (say) metric like so: app_version_updated{instance="eu99",version="1.5.0-abcdefg"} I've tried a number…
kmoe
  • 1,963
  • 2
  • 15
  • 27
58
votes
6 answers

Prometheus - add target specific label in static_configs

I have job definition as follows: - job_name: 'test-name' static_configs: - targets: [ '192.168.1.1:9100', '192.168.1.1:9101', '192.168.1.1:9102' ] labels: group: 'development' Is there any way to annotate targets with…
Krzysztof Rosiński
  • 780
  • 1
  • 8
  • 9
55
votes
3 answers

How can I group labels in a Prometheus query?

If I have a metric with the following labels: my_metric{group="group a"} 100 my_metric{group="group b"} 100 my_metric{group="group c"} 100 my_metric{group="misc group a"} 1 my_metric{group="misc group b"} 2 my_metric{group="misc group c"} …
checketts
  • 14,167
  • 10
  • 53
  • 82
55
votes
5 answers

How to calculate containers' cpu usage in kubernetes with prometheus as monitoring?

I want to calculate the cpu usage of all pods in a kubernetes cluster. I found two metrics in prometheus may be useful: container_cpu_usage_seconds_total: Cumulative cpu time consumed per cpu in seconds. process_cpu_seconds_total: Total user and…
Haoyuan Ge
  • 3,379
  • 3
  • 24
  • 40
50
votes
3 answers

How can I 'join' two metrics in a Prometheus query?

I am using the consul exporter to ingest the health and status of my services into Prometheus. I'd like to fire alerts when the status of services and nodes in Consul is critical and then use tags extracted from Consul when routing those alerts. I…
Rob Best
  • 501
  • 1
  • 4
  • 3
46
votes
5 answers

Increasing Prometheus storage retention

I have Prometheus server installed on my AWS instance but the data is been removed automatically after 15 days. I need to have data for a year or months. Is there anything I need to change in my prometheus configuration? Or do I need any extensions…
Rohit Bharati
  • 563
  • 1
  • 4
  • 5
44
votes
11 answers

Context Deadline Exceeded - prometheus

I have Prometheus configuration with many jobs where I am scraping metrics over HTTP. But I have one job where I need to scrape the metrics over HTTPS. When I access: https://ip-address:port/metrics I can see the metrics. The job that I have added…
xmlParser
  • 1,903
  • 4
  • 19
  • 48
44
votes
4 answers

Prometheus - Convert cpu_user_seconds to CPU Usage %?

I'm monitoring docker containers via Prometheus.io. My problem is that I'm just getting cpu_user_seconds_total or cpu_system_seconds_total. How to convert this ever-increasing value to a CPU percentage? Currently I'm…
M156
  • 1,044
  • 1
  • 12
  • 29
43
votes
2 answers

What does the "instant" checkbox in Grafana graphs based on prometheus do?

I have no clue what the option "instant" means in Grafana when creating graph with Prometheus. Any ideas?
eventhorizon
  • 2,977
  • 8
  • 33
  • 57
40
votes
1 answer

How do I write an "or" logical operator on Prometheus or Grafana

I need to write a query that use any of the different jobs I define. {job="traefik" OR job="cadvisor" OR job="prometheus"} Is it possible to write logical binary operators?
Asier Gomez
  • 6,034
  • 18
  • 52
  • 105
1
2 3
…
99 100