Prometheus - add target specific label in static_configs

Question

I have job definition as follows:

  - job_name: 'test-name'
    static_configs:
      - targets: [ '192.168.1.1:9100', '192.168.1.1:9101', '192.168.1.1:9102' ]
        labels:
          group: 'development'

Is there any way to annotate targets with labels? For instance, I would like to add 'service-1' label to '192.168.1.1:9100', 'service-2' to '192.168.1.1:9101' etc.

upvoted because I didn't find any reference to this `labels` option in the scrap config [documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config), am I alone ? :) — Kuruwan, Mar 18 '21 at 18:03

score 71 · Accepted Answer · answered Apr 16 '19 at 03:32

71

I have the same question before. Here is my solution:

use job_name as the group label
add more target option to separate instance and add labels

For you the code may like this:

  - job_name: 'development'
      static_configs:
      - targets: [ '192.168.1.1:9100' ]
        labels:
          service: '1'
      - targets: [ '192.168.1.1:9101' ]
        labels:
          service: '2'

answered Apr 16 '19 at 03:32

pan congwen

726
6
5

10

\*facepalm* I hadn't realized I could repeat targets and labels were specific to each like that.. Thanks a lot! – Keilaron Apr 24 '19 at 16:38

score 36 · Answer 2 · edited Jul 07 '21 at 21:05

36

Can be like this

  - job_name: 'node'
    static_configs:
    - targets: ['192.168.1.117:9100']
      labels:
        instance: 'linux-ina'
    - targets: ['192.168.1.138:9100']
      labels:
        instance: 'linux-inb'

Tag name can be replaced with instance

edited Jul 07 '21 at 21:05

GG.

21,083
14
84
130

answered Jun 26 '19 at 04:12

zxf曾爷

483
5
5

brian-brazil · Answer 3 · 2018-04-14T23:12:37.543

2

For different services you should usually vary the job label, so I would suggest duplicating the scrape config with a job_name of service1 for one and service2 for the other.

edited Apr 14 '18 at 23:12

answered Apr 14 '18 at 08:32

brian-brazil

31,678
6
93
86

1

Ok, that makes sense but then it may be hard to manage multiple systems. I'd like to have two systems which will have the same services installed, the service names will be then duplicated: - job_name: 'service1' static_configs: - targets: [ '192.168.1.1:9100' ] - job_name: 'service2' static_configs: - targets: [ '192.168.1.1:9101' ] - job_name: 'service1' static_configs: - targets: [ '192.168.1.2:9100' ] - job_name: 'service2' static_configs: - targets: [ '192.168.1.2:9101' ] What is the best solution for this, please? – Krzysztof Rosiński Apr 14 '18 at 08:46
3

https://www.robustperception.io/using-json-file-service-discovery-with-prometheus/ I think this is the solution which will work well for me. It seems that you've also written this blog post, thanks a lot! :-) – Krzysztof Rosiński Apr 14 '18 at 10:39
1

Brian's answer should be considered as correct. Imagine situation if you decide to relabel metrics for single target, using file_sd it is impossible without hacks and overcomplicating things. Anyone can say that you can use honor_labels, or temporal label for further rewriting but it is all dirty hacks you should avoid. – Alexey Antipin Nov 26 '21 at 10:31

score 1 · Answer 4 · answered Jan 21 '19 at 10:29

1

You can use File Based ServiceDiscovery to achieve this. See this Blog Post for more details.

answered Jan 21 '19 at 10:29

Nils Schmidt

3,702
6
23
28

score 1 · Answer 5 · answered Jan 28 '19 at 17:54

You can find information in here - Prometheus Good Config

But I have used these and it worked


  - job_name:  'PostgreSQL-exporter'
    scrape_interval: 60s
    scrape_timeout: 60s
    static_configs:
      - targets: ['localhost:9187']
      - labels:
          name: value-for-the-name

score 1 · Answer 6 · answered Aug 29 '23 at 06:11

The targets do not have to be simple ip:port pairs, you can include extra info and use re-labelling to split out the ip and port from a nice instance name:

scrape_configs:
  - job_name: 'prometheus_metrics'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter_metrics'
    scrape_interval: 5s
    static_configs:
      - targets: ["157.xxx.xxx.xxx:9100@db56a", "176.xxx.xxx.xxx:9100@db56b"]
        labels:
          role: 'postgres_ha'
      - targets: ["46.xxx.xxx.xxx:9100@p1a", "176.xxx.xxx.xxx:9100@p1b", "136.xxx.xxx.xxx:9100@p1c"]
        labels:
          role: 'pulsar'
    relabel_configs:
      - source_labels: [ __address__ ]
        regex: '(.*)@(.*)'
        replacement: $2
        target_label: instance
      - source_labels: [ __address__ ]
        regex: '(.*)@(.*)'
        replacement: $1
        target_label: __address__

Prometheus - add target specific label in static_configs

6 Answers6

Linked