Prometheus alertmanager send notifications to multiple slack channel

Question

We have two different teams working on different applications.I would like send alert notifications to different slack channels via using same alerts expressions. I found some examples but not understand what is the main reason to use receiver: 'default' when try to add new route? What is the role of this and what if it affects if ı change this?

Meanwhile will be appreciate if you can help how should I send the notifations to multiple slack channels.. New one is what I tried.

Current alertmanager.yml

receivers:
  - name: 'team-1'
    slack_configs:
    - api_url: 'https://hooks.slack.com/services/1'
      channel: '#hub-alerts'
route:
  group_wait: 10s
  group_interval: 5m
  receiver: 'team-1'
  repeat_interval: 1h
  group_by: [datacenter]

New alertmanager.yml

alertmanager.yml:
    receivers:
      - name: 'team-1'
        slack_configs:
        - api_url: 'https://hooks.slack.com/services/1'
          channel: '#channel-1'
          send_resolved: true
      
      - name: 'team-2'
        slack_configs:
        - api_url: 'https://hooks.slack.com/services/2'
          channel: '#channel-2'
          send_resolved: true

route:
  group_wait: 10s
  group_interval: 5m
  repeat_interval: 1h
  group_by: [datacenter]
  receiver: 'default'
  routes:
  - receiver: 'team-1'
  - receiver: 'team-2'

score 22 · Accepted Answer · edited Mar 02 '21 at 18:27

You need to set the continue property on your route to true. By default it is false.

The default behaviour of AlertManager is to traverse your routes for a match and exit at the first node it finds a match at.

What you want to do is fire an alert at the match and continue to search for other matches and fire those too.

Relevant documentation section: https://prometheus.io/docs/alerting/latest/configuration/#route

An example using this: https://awesome-prometheus-alerts.grep.to/alertmanager.html

In-lined the example above in case it ever breaks.

# alertmanager.yml

route:
  # When a new group of alerts is created by an incoming alert, wait at
  # least 'group_wait' to send the initial notification.
  # This way ensures that you get multiple alerts for the same group that start
  # firing shortly after another are batched together on the first
  # notification.
  group_wait: 10s

  # When the first notification was sent, wait 'group_interval' to send a batch
  # of new alerts that started firing for that group.
  group_interval: 5m

  # If an alert has successfully been sent, wait 'repeat_interval' to
  # resend them.
  repeat_interval: 30m

  # A default receiver
  receiver: "slack"

  # All the above attributes are inherited by all child routes and can
  # overwritten on each.
  routes:
    - receiver: "slack"
      group_wait: 10s
      match_re:
        severity: critical|warning
      continue: true

    - receiver: "pager"
      group_wait: 10s
      match_re:
        severity: critical
      continue: true

receivers:
  - name: "slack"
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/XXXXXXXXX/XXXXXXXXX/xxxxxxxxxxxxxxxxxxxxxxxxxxx'
        send_resolved: true
        channel: 'monitoring'
        text: "{{ range .Alerts }}<!channel> {{ .Annotations.summary }}\n{{ .Annotations.description }}\n{{ end }}"

  - name: "pager"
    webhook_config:
      - url: http://a.b.c.d:8080/send/sms
        send_resolved: true

score 0 · Answer 2 · answered Nov 01 '22 at 10:52

I have alertmanagerconfig with below configuration and now I need to point info alerts to a null receiver, can I have multiple receiver and receivers?

kind: AlertmanagerConfig
metadata:
  name: Prometheus-alertmanager-config
  namespace: Prometheus
spec:
  route:
    receiver: alert-email-pagerduty-config
    groupBy: ['alertname', 'priority','severity']
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 15m
    continue: true
  receivers: 
  - name: alert-email-pagerduty-config
    emailConfigs:
    - to: {{.to_email}}
      sendResolved: true
      from: {{.from_email}}
      smarthost:  {{.smarthost}}
      authUsername: {{.mail_username}}
      authPassword:
        name: 'alert-smtp-password'
        key: 'password'
      requireTLS: true
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: alert-smtp-password
  namespace: prometheus
stringData:
  password: {{.mail_password}}

score 0 · Answer 3 · answered Jul 08 '23 at 10:14

Note! Make sure there is not any gap or newline between continue: true and next receiver.

# alertmanager.yml

route:
  # When a new group of alerts is created by an incoming alert, wait at
  # least 'group_wait' to send the initial notification.
  # This way ensures that you get multiple alerts for the same group that start
  # firing shortly after another are batched together on the first
  # notification.
  group_wait: 10s

  # When the first notification was sent, wait 'group_interval' to send a batch
  # of new alerts that started firing for that group.
  group_interval: 30s

  # If an alert has successfully been sent, wait 'repeat_interval' to
  # resend them.
  repeat_interval: 30m

  # A default receiver
  receiver: "one"

  # All the above attributes are inherited by all child routes and can
  # overwritten on each.
  routes:
    - receiver: "one"
      group_wait: 10s
      match_re:
        severity: "Critical|High|Medium|Low|warning|info"
      continue: true
    - receiver: "two"
      group_wait: 10s
      match_re:
        severity: "Critical|High|Medium"
      continue: true
    - receiver: "three"
      group_wait: 10s
      match_re:
        severity: "Critical|High"
      continue: true

receivers:
  - name: "one"
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T2ND83CHL/B05G3RPUE03/kEFGLfmR0RtDVEsFUwrlkKv'
        send_resolved: true
        channel: 'one'
        icon_url: https://prometheus.io/assets/prometheus_logo_grey.svg
        title: |-
          [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
          {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
            {{" "}}(
            {{- with .CommonLabels.Remove .GroupLabels.Names }}
              {{- range $index, $label := .SortedPairs -}}
                {{ if $index }}, {{ end }}
                {{- $label.Name }}="{{ $label.Value -}}"
              {{- end }}
            {{- end -}}
            )
          {{- end }}
        text: >-
          {{ range .Alerts -}}
          *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
          *Description:* {{ .Annotations.description }}
          *Details:*
            {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
            {{ end }}
          {{ end }}

  - name: "two"
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T2ND83kuL/B05GGJEAV8R/LEEkDn24LE7BAVQr4HNWW76h'
        send_resolved: true
        channel: 'two'
        icon_url: https://prometheus.io/assets/prometheus_logo_grey.svg
        title: |-
          [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
          {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
            {{" "}}(
            {{- with .CommonLabels.Remove .GroupLabels.Names }}
              {{- range $index, $label := .SortedPairs -}}
                {{ if $index }}, {{ end }}
                {{- $label.Name }}="{{ $label.Value -}}"
              {{- end }}
            {{- end -}}
            )
          {{- end }}
        text: >-
          {{ range .Alerts -}}
          *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
          *Description:* {{ .Annotations.description }}
          *Details:*
            {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
            {{ end }}
          {{ end }}

  - name: "three"
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/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'
        send_resolved: true
        channel: 'three'
        icon_url: https://prometheus.io/assets/prometheus_logo_grey.svg
        title: |-
          [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
          {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
            {{" "}}(
            {{- with .CommonLabels.Remove .GroupLabels.Names }}
              {{- range $index, $label := .SortedPairs -}}
                {{ if $index }}, {{ end }}
                {{- $label.Name }}="{{ $label.Value -}}"
              {{- end }}
            {{- end -}}
            )
          {{- end }}
        text: >-
          {{ range .Alerts -}}
          *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
          *Description:* {{ .Annotations.description }}
          *Details:*
            {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
            {{ end }}
          {{ end }}

Prometheus alertmanager send notifications to multiple slack channel

3 Answers3