Prometheus Docker fails to start with `Template: (dynamic): parse: template: :10: undefined variable "$labels"`

Question

I am trying to run Prometheus on Nomad. Everything things look fine but when I add an alert rule with templating in annotations it fails.

here is a simple alert rule:

groups:
- name: alertmanager.rules
  rules:
  - alert: AlertmanagerDown
    expr: up{job="alertmanager"} == 0
    for: 1s
    labels:
      severity: critical
    annotations:
      description: 'Alertmanager on {{ $labels.role }} instance {{ $labels.node }} has not produced any metrics for 5 minutes'
      summary: 'Alertmanager is down on {{ $labels.role }} instance {{ $labels.node }}'

Container fails with Template: (dynamic): parse: template: :10: undefined variable "$labels"

If I remove annotations section it starts without any issue.

Answer 1

Had to set left_delimiter and right_delimiter to something other than {{ and }} https://www.nomadproject.io/docs/job-specification/template.html#left_delimiter

Answer 2

I met the same issue when deploying Prometheus. Deploying Prometheus via a helm chart will have an issue as the same delimiters ({{ and }}) are used both by helm templating and the alerting templating in Prometheus.

For my case, the solution is:

  recordType: ALARM
  managedObjectInstance: "Namespace={{ "{{" }} $labels.app_kubernetes_io_instance }},Instance={{ "{{" }} $labels.instance }}"

More example can be found here.