4

We want to get all metric names from Prometheus server filtered by a particular label.

Step 1 : Used following query to get all metric names, query succeeded with all metric names.

curl -g 'http://localhost:9090/api/v1/label/__name__/values

Step 2 : Used following query to get all metrics names filtered by label, but query still returned all metric names.

curl -g 'http://localhost:9090/api/v1/label/__name__/values?match[]={job!="prometheus"}'

Can somebody please help me filter all metric names by label over http? Thanks

curl -G -XGET http://localhost:9090/api/v1/label/__name__/values --data-urlencode 'match[]={__name__=~".+", job!="prometheus"}'

@anemyte, Still returns all the results. Can you please check the query

user3082802
  • 47
  • 1
  • 1
  • 6
  • the last one certainly works for me: https://pastebin.com/izvkhJpP – anemyte Dec 13 '21 at 07:20
  • @anemyte, still get all the metrics even though I specify different labels. I also get same number of metrics through https://github.com/prometheus/client_golang api. // LabelValues performs a query for the values of the given label, time range and matchers. LabelValues(ctx context.Context, label string, matches []string, startTime time.Time, endTime time.Time) (model.LabelValues, Warnings, error) – user3082802 Dec 23 '21 at 13:28
  • What other label selectors have you tried? Does it work with something dumb simple, like `match[]=my_metric_foo`? – anemyte Dec 23 '21 at 15:54
  • @anemyte, it always returns all the metric names irrespective of any match[] from the prometheus server. – user3082802 Dec 23 '21 at 16:04
  • Well at this point I can only tell that it is either a bug, or your `match[]` parameter is not recognized at all. Check logs, try requesting labels in browser (eliminates problems with curl) _**and**_ composing the selector in graph panel. – anemyte Dec 24 '21 at 06:21
  • @anemyte, https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers using this api I am able to filter and fetch time-series. But it contains all the labels associated with the metric name too. – user3082802 Dec 24 '21 at 11:05
  • @anemyte, prometheus server version is : 2.22.0, is there any special configuration in your prometheus.yml file ? – user3082802 Dec 24 '21 at 11:19
  • Indeed, it doesn't work in v2.22.0. Matcher support was added to this API in v2.24.0 ([release](https://github.com/prometheus/prometheus/releases/tag/v2.24.0), [pr](https://github.com/prometheus/prometheus/pull/8301)) – anemyte Dec 24 '21 at 13:43
  • @anemyte, Perfect I upgraded prometheus server to v2.25.0 and it works like that. Thanks! – user3082802 Dec 26 '21 at 11:12

2 Answers2

7

Although this seems simple at the first glance, it turned out to be a very tricky thing to do.

  1. The match[] parameter and its value have to be encoded. curl can do that with --data-urlencode argument.

  2. The encoded match[] parameter must be present in the URL and not in application/x-www-form-urlencoded header (where curl puts the encoded value by default). Thus, the -G (the capital one!) key is also required.

  3. {job!="prometheus"} isn't a valid query. It gives the following error:

    parse error: vector selector must contain at least one non-empty matcher

    It is possible to overcome with this inefficient regex selector: {__name__=~".+", job!="prometheus"}. It would be better to replace it with another selector if possible (like {job="foo"}, for example).

Putting all together:

curl -XGET -G 'http://localhost:9090/api/v1/label/__name__/values' \
  --data-urlencode 'match[]={__name__=~".+", job!="prometheus"}'

Using selectors as in the example above became possible since Prometheus release v2.24.0.

anemyte
  • 17,618
  • 1
  • 24
  • 45
  • This will give all metrics and their values with the attached label. Our program runs in memory constrained environment, so wanted to understand what is the problem with the query that I posted. Is there any issue with the query or any bug on the Prometheus end ? – user3082802 Dec 13 '21 at 04:29
  • @user3082802 Oh, sorry, dunno what happened to me to write that. See the query above I posted, it works for me. – anemyte Dec 13 '21 at 06:02
  • I used the exact query that you posted above, but get following error. "Method Not Allowed" – user3082802 Dec 13 '21 at 06:13
  • @user3082802 replace `-g` with `-XGET` – anemyte Dec 13 '21 at 06:16
  • curl -X GET http://localhost:9090/api/v1/label/__name__/values --data-urlencode 'match[]={job!="prometheus"}' and it returned all the metrics, so {job="prometheus"} and {job!="prometheus"} return exact same result – user3082802 Dec 13 '21 at 06:18
  • anemyte, are you able to filter out results based on the query that you gave, I see exact same results for both the queries? which is not correct – user3082802 Dec 13 '21 at 06:28
  • @user3082802 you're right. Turns out there are two problems with the command. 1. `curl` adds the `match[]` parameter as `application/x-www-form-urlencoded` header and Prometheus seems to ignore it at this endpoint. Adding `-G` (the big one) solve this problem. 2. Prometheus does not allow to use only negative selector this way, the query has to be adjusted as well: `'match[]={__name__=~".+", job!="prometheus"}'` – anemyte Dec 13 '21 at 07:06
  • This didn't work for me. The label/value seems just got ignored. – Jianwu Chen Aug 15 '23 at 01:10
0

The accepted answer doesn't work for me. The match[] parameter takes no effect. It will always return all the metrics names. There's no proper doc and example in the official doc as well. Seems it's just not proper implemented.

Struggled for a while. Got my solution with promql:

group ({k8s_namespace="someName", app="someApp"}) by (__name__)

With curl:

curl -v http://host:port/api/v1/query --data-urlencode 'query=group({k8s_namespace="someName", app="someApp"}) by (__name__)'
Jianwu Chen
  • 5,336
  • 3
  • 30
  • 35