4

I used this alret

        - alert: my alert
          expr: status{status="ERROR"}
          for: 30m
          labels:
            severity: WARNING   
          annotations:
            myData: "{{ $labels.myData }}"
            myData2: "{{ $labels.myData2 }}"

I got an error ERROR - templates/: parse error in "prometheus/templates/alertmanager-prometheusRule.yaml": template: prometheus/templates/alertmanager-prometheusRule.yaml:419: undefined variable "$labels"

I saw the same issue in

Prometheus Docker fails to start with `Template: (dynamic): parse: template: :10: undefined variable "$labels"`

but I didn't understand how to solve it

in the configuration I used this data

text: "{{ range .Alerts -}}{{ .Annotations.myData }}{{ .Annotations.myData2}}{{ end-}}"

The error is from helm lint

user1365697
  • 5,819
  • 15
  • 60
  • 96
  • What happens if you change: ```myData: "{{ $labels.myData}}"``` To: ```description: "{{ $labels.myData}}"``` – Saf Nov 05 '20 at 08:50
  • @Saf - I have two annotation in my keys . I will update my question . did you mean to change only the myData to description. but if I have two annotations what I can do. also I get this data in the text like Annotations.myData – user1365697 Nov 05 '20 at 08:52
  • @saf - in the answer of the question he wrote 0 Had to set left_delimiter and right_delimiter what does it mean ? – user1365697 Nov 05 '20 at 08:53
  • Not a clue on the left_delimiter stuff, it doesn't make sense to me. Do you have any other working alerts that use the format "myData: "<>" ? I've never seen that before but that doesn't mean it doesn't work, just my lack of knowledge. – Saf Nov 05 '20 at 08:56

4 Answers4

17

It seems you are deploying your Prometheus setup via a helm chart. This causes an issue as the same delimiters ({{ and }}) are used both by helm templating and the alerting templating in Prometheus.

The {{ $labels.myData }} has to reach prometheus config intact, so helm must not process it.

The simplest way would be to use:

{{ "{{" }} $labels.myData }}

The {{ "{{" }} block will be processed by helm and will produce {{ as a result with rest of the line not being altered and will get you the result you need.

bjakubski
  • 1,477
  • 10
  • 9
1

You can use this awenser

OR

you can simply instead of using double qoutes "", use the single-qoutes ''

e.g. here would be like:

annotations:
  myData: '{{ $labels.myData }}'
  myData2: '{{ $labels.myData2 }}'

source: Prometheus Docs

Ali
  • 922
  • 1
  • 9
  • 24
  • This will not work, as the issue here is that helm will intercept and process it before it reaches prometheus config – bjakubski May 24 '22 at 10:54
0

Here is a python script to make @bjakubski's change in bulk on a file:


import sys
import re

f = sys.argv[1]

regex = re.compile(r"(\{\{.*?\}\})")

with open (f, "r") as myfile:
     s=myfile.read()

ret = regex.sub(lambda x:x.group(1)[0:2]+' "{{" }}'+ x.group(1)[2:],s)

print ret
chrismead
  • 2,163
  • 3
  • 24
  • 36
0

Kind of the same answer as @bjakubski, but with examples:

I would suggest doing the same way as the official Prometheus helm chart is doing to pass alerts:

description: Filesystem on {{`{{`}} $labels.device {{`}}`}} at {{`{{`}} $labels.instance {{`}}`}} has only {{`{{`}} printf "%.2f" $value {{`}}`}}% available space left and is filling up.

They have plenty of examples in the repository.

Link: https://github.com/prometheus-community/helm-charts/blob/d6c45e97eca55e6212ef8acf546b45aa7851c72e/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/node-exporter.yaml

François
  • 1,793
  • 14
  • 19