Stack Overflow
Stack Overflow

Questions tagged [prometheus-alertmanager]

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.

Source code is available here: https://github.com/prometheus/alertmanager

753 questions
17
votes
1 answer

Prometheus Alert Manager: How do I prevent grouping in notifications

I'm trying to setup Alert Manager in a simple setup where it would send one Slack notification for each notification in receives. I've hoped to disable grouping by removing the group_by configuration. The problem is, that when I send 2 alert one…
Yehuda Adler
  • 171
  • 1
  • 1
  • 4
16
votes
2 answers

scrape interval and evaluation interval in prometheus

My scrape interval and evaluation interval are way off from each other as whown below (15s vs 4m). When I feed metrics to the endpoint, I find that the rules are evaluated every 4m which is expected. However, what I dont understand is that it does…
TheMonkWhoSoldHisCode
  • 2,182
  • 3
  • 26
  • 40
15
votes
4 answers

How to silence Prometheus Alertmanager using config files?

I'm using the official stable/prometheus-operator chart do deploy Prometheus with helm. It's working good so far, except for the annoying CPUThrottlingHigh alert that is firing for many pods (including the own Prometheus' config-reloaders…
Eduardo Baitello
  • 10,469
  • 7
  • 46
  • 74
15
votes
3 answers

Multiple Targets on prometheus

I've configured prometheus on Centos, version details are follows. prometheus-2.5.0.linux-386 I've added two targets on the prometheus.yml configuration file, all the servers node exporters are running. Config as follows, scrape_configs: -…
user10590355
14
votes
4 answers

How to send alert for every error in my logs using Promtail / Loki - AlertManager?

I'm using Promtail + Loki to collect my logs and I can't figure how can I alert for every error in my log files. I'm also using Prometheus, Alertmanager and Grafana. I've seen some people have managed to achieve that, but none of them explained the…
EnTm
  • 163
  • 1
  • 1
  • 6
14
votes
3 answers

How to set Prometheus Alertmanager external URL via configuration

I'm using a vanilla Docker container to start an Alertmanager. As far as I know, I cannot provide the external URL via parameter in this case, so I have to find another way. Is it possible to set the URL via configuration file or environment…
eventhorizon
  • 2,977
  • 8
  • 33
  • 57
12
votes
2 answers

Prometheus query and case sensitivity

I have one query where I am trying to join two metrics on a label. K_Status_Value == 5 and ON(macAddr) state_details{live="True"} The label macAddr is present in both the metrics. The value of the label appears in 'K_Status_Value' sometimes in upper…
Arnav Bose
  • 791
  • 4
  • 13
  • 27
12
votes
2 answers

Alert when docker container pod is in Error or CarshLoopBackOff kubernetes

I have my kubernetes cluster setup on AWS where I am trying to monitor several pods, using cAdvisor + Prometheus + Alert manager. What I want to do it launch an email alert (with service/container name) if a container/pod goes down or stuck in Error…
shiv455
  • 7,384
  • 19
  • 54
  • 93
11
votes
7 answers

How to snooze prometheus alert for specific time

I have faced some issues with Prometheus memory alert. If I take the backup of Gitlab then memory usage going up to 95%. I want to snooze memory alert for a specific time. e.g. If I am taking a backup at 2 AM then I need to snooze Prometheus memory…
Abhijit
  • 111
  • 1
  • 1
  • 3
11
votes
3 answers

how to install latest version of prometheus/promtool in ubuntu?

I downloaded my prometheus version is 2.3.2 wget https://github.com/prometheus/prometheus/releases/download/v2.3.2/prometheus-2.3.2.linux-amd64.tar.gz untared and prometheus already running as service. I want to verify my Prometheus alert manager…
anand babu
  • 335
  • 1
  • 3
  • 13
10
votes
4 answers

Can't load prometheus.yml config file with docker (prom/prometheus)

I am trying to load prometheus with docker using the following custom conf file: danilo@machine:/prometheus-data/prometheus.yml: global: scrape_interval: 15s # By default, scrape targets every 15 seconds. # Attach these labels to any time…
Danilo Caetano
  • 209
  • 2
  • 3
  • 9
9
votes
3 answers

Prometheus alertmanager send notifications to multiple slack channel

We have two different teams working on different applications.I would like send alert notifications to different slack channels via using same alerts expressions. I found some examples but not understand what is the main reason to use receiver:…
semural
  • 3,583
  • 8
  • 37
  • 67
9
votes
2 answers

Prometheus query to average over time by a specific label

I need to query a metric and find out the average value of the metric over a period of 24hrs. But using using avg_over_time directly on the metric won't work. There is a specific ipaddr label. The average has to be grouped by each ipaddr. Now,…
Arnav Bose
  • 791
  • 4
  • 13
  • 27
8
votes
3 answers

Prometheus Alert Rule for Absent Discovered Target

I got an alert while configuring the monitoring module using prometheus/kube-prometheus-stack 25.1.0. Alert [FIRING:1] KubeProxyDown - critical Alert: Target disappeared from Prometheus target discovery. - critical Description: KubeProxy has…
Boney Jacob
  • 111
  • 1
  • 6
8
votes
3 answers

How to detect a new metrics with Prometheus alerting rule

Say I have a metrics request_failures for users. For each user I add a unique label value to the metrics. So for user u1, when a request failed twice, I get the following metrics: request_failures{user_name="u1"} 2 I also have a rule that fires…
Jay Xue
  • 81
  • 1
  • 3
1
2 3
…
50 51