helm prometheus-operator not getting custom metrics from api

Question

I'm using the helm prometheus-operator chart: https://github.com/helm/charts/blob/master/stable/prometheus-operator/values.yaml and I expected it to get my custom metrics from my golang api as I did previously by "hardcoding" the name of the service and the port in the values.yml file:

 scrape_configs:
    - job_name: 'custom-api'
      static_configs:
        - targets: ['custom-api-service.backend.svc.cluster.local:8000']

However, as I have more microservices I know that it can also be done dynamically using the _meta tags. Example: __meta_kubernetes_service_name

However, I haven't figure it out what should I modify from the values.yaml file to make it work.

Grafana is getting my cpu and memory usage from the custom-api but custom-api is not appearing in the targets tab from the prometheus dashboard which is weird...

These are my services:

apiVersion: v1
kind: Service
metadata:
  name: custom-api-service
  namespace: backend
  labels:
    service: custom-api-service
spec:
  type: NodePort
  ports:
    - port: 8000
      targetPort: 8000
      nodePort: 30080
      protocol: TCP
      name: custom-api
  selector:
    component: goapi

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: servicemonitor
  namespace: backend
  labels:
    service: servicemonitor
spec:
  selector:
    matchLabels:
      service: custom-api-service
  endpoints:
  - port: custom-api

score 1 · Answer 1 · answered Apr 12 '20 at 13:30

You will have to create a Service monitor CRD to scrape your metrics.

Let's say you have a k8s service (here: example-app) which is used to communicate with your microservices. Make sure that your microservice exposes Prometheus metrics at a certain port and the k8s service also includes that port (here: prom).

kind: Service
apiVersion: v1
metadata:
  name: example-app
  labels:
    app: example-app
spec:
  selector:
    app: example-app
  ports:
  - name: prom
    port: 8080
  - name: other-port
    port: xxxx

This Service object is discovered by a ServiceMonitor, which selects in the same way. You need to make sure that the matchLabels of serviceMonitor object matchs the metadata.labels of the service.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-app
  labels:
    team: frontend
spec:
  selector:
    matchLabels:
      app: example-app
  endpoints:
  - port: prom

Once you have created the serviceMonitor object, the operator controller will do the rest for you (ie. update the Prometheus configuration). You can also provide custom configuration via serviceMonitor object.

For more details visit Getting started with Prometheus operator.

they are in different namespaces. Helm prometheus-operator is inside monitoring namespace and my `custom-api` is inside the backend namespace how can I configure your `ServiceMonitor ` accordingly? — alex, Apr 12 '20 at 14:09
I updated my question with my current service monitor configuration @KamolHasan — alex, Apr 12 '20 at 14:23
@alex you can use the `namespaceSelector` on the [ServiceMonitor resource CRD](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/design.md#servicemonitor) to specify the namespace. In my case I wanted it to pick up all namespaces so I just set to `any`. ` spec: namespaceSelector: any: true ` I also found it important to check the `Prometheus` CRD. For me it had the following ` serviceMonitorSelector: matchLabels: release: prometheus-stack ` which meant I had to have a `prometheus-stack` release label for my CRD — Samuel Garratt, Nov 04 '21 at 09:56

Arghya Sadhu · Answer 2 · 2020-04-12T16:36:48.533

1

The Prometheus resource includes a field called serviceMonitorSelector, which defines a selection of ServiceMonitors to be used. By default and before the version v0.19.0, ServiceMonitors must be installed in the same namespace as the Prometheus instance. With the Prometheus Operator v0.19.0 and above, ServiceMonitors can be selected outside the Prometheus namespace via the serviceMonitorNamespaceSelector field of the Prometheus resource

In the monitoring namespace create a Prometheus object which selects the ServiceMonitor by label service: servicemonitor

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
spec:
  serviceAccountName: <service-account-name>
  serviceMonitorSelector:
    matchLabels:
      service: servicemonitor
  resources:
    requests:
      memory: 400Mi
  enableAdminAPI: false

The serviceAccountName you can find out in monitoring namespace as helmreleasename-prometheus-operator-prometheus

edited Apr 12 '20 at 16:36

answered Apr 12 '20 at 14:26

Arghya Sadhu

41,002
9
78
107

which one? there are at least 20 configmaps added by default in that namespace when I install the chart – alex Apr 12 '20 at 14:43
can you add it in the values.yaml file – Arghya Sadhu Apr 12 '20 at 15:39
it didn't work and the answer should have been the same as this other thread but it didn't work either: https://stackoverflow.com/questions/60706343/prometheus-operator-enable-monitoring-for-everything-in-all-namespaces – alex Apr 12 '20 at 15:52
create a Prometheus object as in the answer – Arghya Sadhu Apr 12 '20 at 16:09

helm prometheus-operator not getting custom metrics from api

2 Answers2