Can't access default k3s metrics server from Prometheus-Adapter

Question

I have installed Prometheus-adapter along with the default metrics-server that comes with k3s securely on port 443.

Unfortunately, I get no resources when I query custom.metrics.k8s.io

$  kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": []
}

When I look at the logs of Prometheus-adapter I get unable to update list of all metrics: unable to fetch metrics for query ...: x509: certificate is valid for localhost, localhost, not metrics-server.kube-system

How can I resolve this issue?

have you installed Prometheus, Prometheus Operator and Prometheus Adapter thru Helm or YAML ? — jmvcollaborator, Oct 10 '22 at 06:39
@jmvcollaborator https://github.com/k3s-io/k3s/tree/master/manifests/metrics-server these are the resources k3s metrics server has — realsarm, Oct 10 '22 at 07:26
Unfortunatly I'm still getting same error `unable to update list of all metrics: unable to fetch metrics for query ...: x509: certificate is valid for localhost, localhost, not metrics-server.kube-system` @jmvcollaborator — realsarm, Oct 12 '22 at 06:57

score 1 · Accepted Answer · answered Oct 19 '22 at 06:51

To solve this issue, I had to create separate certificate for both metrics-server and adapter. Adapter also has an issue about adding capability to ignore cert validation which wasn't merged.

For metrics-server and cert request I used the following:

{
  "hosts": [
    "prometheus-adapter",
    "prometheus-adapter.monitoring",
    "prometheus-adapter.monitoring.svc",
    "prometheus-adapter.monitoring.pod",
    "prometheus-adapter.monitoring.svc.cluster.local",
    "prometheus-adapter.monitoring.pod.cluster.local",
    "<pod ip>",
    "<service ip>"
  ],
  "CN": "prometheus-adapter.monitoring.pod.cluster.local",
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
}

{
  "hosts": [
    "metrics-server",
    "metrics-server.kube-system",
    "metrics-server.kube-system.svc",
    "metrics-server.kube-system.pod",
    "metrics-server.kube-system.svc.cluster.local",
    "metrics-server.kube-system.pod.cluster.local",
    "<service ip>",
    "<pod ip>"
  ],
  "CN": "metrics-server.kube-system",
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
}

For ca, you can create your certificate authority or use Kubernetes signers as indicated here The only point worth noting here is that if you use either of signers, you should mount the ca bundle yourself to your deployments.

Finally, mount tls keys and ca bundle to your deployment.

  extraArguments:
    - --tls-cert-file=/var/run/serving-cert/tls.crt
    - --tls-private-key-file=/var/run/serving-cert/tls.key
    - --client-ca-file=/etc/ssl/certs/ca.crt

jmvcollaborator · Answer 2 · 2022-10-10T07:49:34.817

0

You have two main options here:

Configure Prometheus to ignore ssl, setting to true the skip verify

tls_config:
insecure_skip_verify: true

Add the certificate to the scrape_configs under tls_config

tls_config: ca_file: /path/to/prometheus.crt

UPDATE:

On metrics-server-deployment.yaml

Leave internalip only, on the --kubelet-preferred-address-type

--kubelet-preferred-address-types=InternalIP

edited Oct 10 '22 at 07:49

answered Oct 10 '22 at 07:10

jmvcollaborator

2,141
1
6
17

Can't access default k3s metrics server from Prometheus-Adapter

2 Answers2