5

I am deploying Prometheus with Thanos with the given config:

global:
  scrape_interval: 15s  # By default, scrape targets every 15 seconds.

 # Attach these labels to any time series or alerts when communicating with
 # external systems (federation, remote storage, Alertmanager).
 external_labels:
   monitor: %%CLUSTER%%
   replica: $(HOSTNAME)

I got this from the Thanos example documents but I am not able to make this work. "%%CLUSTER%% gives "cannot start with the given token" error whereas the HOSTNAME substitution never happens.

I read on this issue that variable substitution is not supported but the Thanos docs says otherwise. https://github.com/prometheus/prometheus/issues/2357

Is there any other way to implement this?

Nicola Ben
  • 10,615
  • 8
  • 41
  • 65
swetad90
  • 784
  • 1
  • 13
  • 34

2 Answers2

2

There is a standard way to do it since Prometheus 2.27.

See disabled_features

--enable-feature=expand-external-labels

Replace ${var} or $var in the external_labels values according to the values of the current environment variables. References to undefined variables are replaced by the empty string.

And this reply by roidelaplui

Louis Go
  • 2,213
  • 2
  • 16
  • 29
1

UPDATE:

Looks like Thanos gives this out of the box. We have to configure Thanos to watch for the config file which has this templated variables and produce the output to a directory. And Prometheus then picks this file as its config file. This file would have the "substituted values" already from Thanos sidecar.

swetad90
  • 784
  • 1
  • 13
  • 34
  • It's unclear what you are trying to achieve but your original question is failing not because those are some special template variables for Thanos but because they are supposed to be replaced by the tutorial setup script at https://github.com/improbable-eng/thanos/blob/e01a5955bf46705012a12a321b688413957a7796/tutorials/kubernetes-demo/setup.sh. – Andy Shinn Jul 12 '19 at 17:02
  • I think Thanos has that set up as part of its sidecar as well. Because, when I give label as '$(HOSTNAME)' and point that config file to the sidecar's reloader config, it automatically substitutes it with the pod name in the output file. Let me know if I am wrong in understanding this! – swetad90 Jul 15 '19 at 15:01