5

The idea here is that I'm considering alerting on a metric that corroborates a metric that another tool is monitoring. For example: I might have a service doing some task by feeding off a queue and when processed tasks drop to zero an alert needs to fire to. Another service is monitoring when a queue hasn't reduced in size, basically detecting from the sender's side, that requests aren't being processed, then it too needs to fire.

However, I'd like to avoid two alerts. Too much noise would effect MTTR. Is there a way I could either avoid the second alert, join the alerts, or mention the other alert in a single alert. The last one is kind of like wrapping an exception/error inside another exception/error, also likely my preferred method if possible.

Is it possible to conditionally alert with Prometheus based on if another is firing or not?

lucidquiet
  • 6,124
  • 7
  • 51
  • 88

1 Answers1

0

Yes.

Inhibition Rules are what you are looking for. You can use this configuration in tandem with carefully selected labels in the set of alerts you are concerned with. It will make sure if an alert is already firing then another matching alert does not.

aaruja
  • 371
  • 2
  • 12