0

I'am trying to set up the metrics to activate HPA (Horizontal Pod Auto-scaling) I follow this tutorial only the Custom Metrics (Prometheus) .

Unfortunately when i execute the command below :

  kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"custom.metrics.k8s.io/v1beta1","resources":[]}  

I must to see a lot of thing on resources however there is nothing.

clxoid
  • 2,577
  • 12
  • 21
morla
  • 675
  • 4
  • 15
  • 32

1 Answers1

1

This might be the issue how you setup metrics-server and metrics-server could not able to find your resources on InternalIP.

The solution is to replace the metrics-server-deployment.yaml file in metrics-server/deploy/1.8+ with the following yaml file:

apiVersion: v1
 kind: ServiceAccount
 metadata:
   name: metrics-server
   namespace: kube-system
 ---
 apiVersion: extensions/v1beta1
 kind: Deployment
 metadata:
   name: metrics-server
   namespace: kube-system
   labels:
     k8s-app: metrics-server
 spec:
   selector:
     matchLabels:
       k8s-app: metrics-server
   template:
     metadata:
       name: metrics-server
       labels:
         k8s-app: metrics-server
     spec:
       serviceAccountName: metrics-server
       volumes:
       # mount in tmp so we can safely use from-scratch images and/or read-only containers
       - name: tmp-dir
         emptyDir: {}
       containers:
       - command:
         - /metrics-server
         - --metric-resolution=30s
         - --kubelet-insecure-tls
         - --kubelet-preferred-address-types=InternalIP
         name: metrics-server
         image: k8s.gcr.io/metrics-server-amd64:v0.3.1
         imagePullPolicy: Always
         volumeMounts:
         - name: tmp-dir
           mountPath: /tmp

Also, enable the --authentication-token-webhook in kubelet.conf, then you will be able to get the metrics.

Also, checkout my answer for step by step instruction to set the HPA using metrics-server.

How to Enable KubeAPI server for HPA Autoscaling Metrics

Hope this helps. Revert back if you face any issues.

Prafull Ladha
  • 12,341
  • 2
  • 37
  • 58
  • thanks for your reply . I apply the modification on metrics-server-deployment.yaml. but know i try to know how to generate the kubelet.conf to set the value – morla Dec 20 '18 at 09:02
  • The kubelet conf should already be there at `/var/lib/kubelet/config.yaml`. Please check for that file. – Prafull Ladha Dec 20 '18 at 09:03
  • i dont see the file on `/var/lib/kubelet` sniff – morla Dec 20 '18 at 09:15
  • How did you setup your cluster? Did you use kubeadm? And which version of kubernetes you're using? – Prafull Ladha Dec 20 '18 at 09:19
  • i found it ! in `/etc/kubernetes/kubelet.conf` – morla Dec 20 '18 at 09:44
  • so i added the params on the file [here](https://gist.github.com/zyriuse75/7f8c4535e35938b97abb7901a752465f) . i restarted `kubectl` . but i still dont have the metrics :( grr.. – morla Dec 20 '18 at 09:49
  • That file is not a kubelet config file, that is certificate file for kubelet. Run the following command: `systemctl status kubelet -l` and share the output. – Prafull Ladha Dec 20 '18 at 09:52
  • You will find below the [result of the command](https://gist.github.com/zyriuse75/e6bff7141af533c9ad128989c8fd68a6) – morla Dec 20 '18 at 10:03
  • Cool . So you have a file `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` You have to edit that file and add `--authorization-mode=Webhook --authentication-token-webhook` the first argument you have add the second one in that and restart the kubelet – Prafull Ladha Dec 20 '18 at 10:11
  • i just see that in the file [10-kubeadm.conf](https://gist.github.com/zyriuse75/c883a14035de79e56a4102657327177e) the params is already set – morla Dec 20 '18 at 10:23
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/185545/discussion-between-prafull-ladha-and-morla). – Prafull Ladha Dec 20 '18 at 10:27
  • i just added as well the following params `--anonymous-auth=false --authentication-token-webhook=true ` – morla Dec 20 '18 at 10:30
  • you dont need to add `=true` in `--authentication-token-webhook`. Just add `--authentication-token-webhook` and reload it using `systemctl daemon-reload` and restart it using `systemctl restart kubelet` – Prafull Ladha Dec 20 '18 at 10:33
  • almost but not yet sniff.. i will get my metric !! :) in the same time i search on internet and i found this post on [github](https://github.com/kubernetes/kops/issues/5176) it looks similar . – morla Dec 20 '18 at 10:40
  • Sure. Enable the webhook and it will work. I have tried it myself. :) – Prafull Ladha Dec 20 '18 at 10:42
  • i check again and nothing happen . however i set the params on the file and it start correctly . `"KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --anonymous-auth=false --authentication-token-webhook --client-ca-file=/etc/kubernetes/pki/ca.crt" ` Did i miss something ? – morla Dec 20 '18 at 10:49
  • Could you please again share the output of `systemctl status kubelet -l` . I want to check the new applied changes – Prafull Ladha Dec 20 '18 at 10:57
  • the output of the [command](https://gist.github.com/zyriuse75/99e5d8d74972d32f83f9a3ce82d224e4) – morla Dec 20 '18 at 11:06
  • Okay. So this is fine. Now you might want to check the logs of `kubectl logs -n kube-system` – Prafull Ladha Dec 20 '18 at 11:25
  • i join the chat – morla Dec 20 '18 at 11:34