How to expose kubernetes metric server api to curl from inside the pod?

Question

I am using metric server to get the usage of my Kubernetes cluster. But in order to use it from outside the host, I need to use "kubectl proxy". But i don't want to do that as it is not intended to run on background. I want it to be run as a service continuously

How can i achieve these

expected output curl clusterip:8001/apis/metrics.k8s.io/v1beta1/nodes

{
  "kind": "NodeMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes"
  },
  "items": [
    {
      "metadata": {
        "name": "manhattan-master",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/manhattan-master",
        "creationTimestamp": "2019-11-15T04:26:47Z"
      },
      "timestamp": "2019-11-15T04:26:33Z",
      "window": "30s",
      "usage": {
        "cpu": "222998424n",
        "memory": "3580660Ki"
      }
    }
  ]

I tried by using LoadBalancig service metrics-server-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    k8s-app: metrics-server
  ports:
  - port: 443
    protocol: TCP
    targetPort: main-port
  externalTrafficPolicy: Local
  type: LoadBalancer

kubectl describe service metrics-master -n kube-system

[root@manhattan-master 1.8+]# kubectl describe service metrics-server -n kube-system
Name:                     metrics-server
Namespace:                kube-system
Labels:                   kubernetes.io/cluster-service=true
                          kubernetes.io/name=Metrics-server
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"kubernetes.io/cluster-service":"true","kubernetes.io/name":"Me...
Selector:                 k8s-app=metrics-server
Type:                     LoadBalancer
IP:                       10.110.223.216
Port:                     <unset>  443/TCP
TargetPort:               main-port/TCP
NodePort:                 <unset>  31043/TCP
Endpoints:                10.32.0.7:4443
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     32208
Events:                   <none>

I used service name "loadbalancer" instead of "clusterIp" in service.yaml file. It will give you loadbalancer endpoint and then you can use it. — Shree Prakash, Nov 18 '19 at 10:03
I tried but i didn't get any ExternalIp in that case and my top command also failing — UDIT JOSHI, Nov 18 '19 at 11:04
Hi, you can use your apiserver fqdn, `curl -kv $API_SERVER_FQDN/apis/metrics.k8s.io/v1beta1/nodes`, that's how kubectl get the information about usage — Suresh Vishnoi, Nov 18 '19 at 11:36
@UDITJOSHI it should be there , just do kubectl get svc -n kube-system or in which namespace you deployed metric server — Shree Prakash, Nov 18 '19 at 16:42
@ShreePrakash I have added the description of my metric-server service. Can you please tell me what wrong am i doing? — UDIT JOSHI, Nov 19 '19 at 05:15
@UDITJOSHI this 10.110.223.216:443 is LoadBalancer endpoint . right ?? — Shree Prakash, Nov 19 '19 at 07:11
Yes. But i didn't get External ip . I also tried with NodePod service but the curl ip:port gives null — UDIT JOSHI, Nov 19 '19 at 07:47

Mark Watney · Accepted Answer · 2019-11-19T13:56:31.043

This is possible by creating a new service to expose the Metrics Server. Your Metrics Server Service should look like this:

apiVersion: v1
kind: Service
metadata:
  labels:
    kubernetes.io/name: Metrics-server-ext
  name: metrics-server-ext
  namespace: kube-system
  selfLink: /api/v1/namespaces/kube-system/services/metrics-server
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: https
  selector:
    k8s-app: metrics-server
  sessionAffinity: None
  type: LoadBalancer

If you try to access this service you will face some problems with authorization and you need to do some things to give all necessary authorizations.

After creating the service you will need to create a Cluster Role Binding so our service can have access to the data:

$ kubectl create clusterrolebinding node-admin-default-svc --clusterrole=cluster-admin --serviceaccount=default:default

Before running curl command we need to get the token so we can pass this on our curl command:

$ TOKEN=$(kubectl get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='default')].data.token}"|base64 --decode)

Get your service external IP:

kubectl get svc/metrics-server-ext -n kube-system -o jsonpath='{..ip}'

Your curl command should pass the Token key to get Authorization:

curl -k https://34.89.228.98/apis/metrics.k8s.io/v1beta1/nodes --header "Authorization: Bearer $TOKEN" --insecure

Sample output:

{
 "kind": "NodeMetricsList",
 "apiVersion": "metrics.k8s.io/v1beta1",
 "metadata": {
   "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes"
 },
 "items": [
   {
     "metadata": {
       "name": "gke-lab-default-pool-993de7d7-ntmc",
       "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/gke-lab-default-pool-993de7d7-ntmc",
       "creationTimestamp": "2019-11-19T10:26:52Z"
     },
     "timestamp": "2019-11-19T10:26:17Z",
     "window": "30s",
     "usage": {
       "cpu": "52046272n",
       "memory": "686768Ki"
     }
   },
   {
     "metadata": {
       "name": "gke-lab-default-pool-993de7d7-tkj9",
       "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/gke-lab-default-pool-993de7d7-tkj9",
       "creationTimestamp": "2019-11-19T10:26:52Z"
     },
     "timestamp": "2019-11-19T10:26:21Z",
     "window": "30s",
     "usage": {
       "cpu": "52320505n",
       "memory": "687252Ki"
     }
   },
   {
     "metadata": {
       "name": "gke-lab-default-pool-993de7d7-v7m3",
       "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/gke-lab-default-pool-993de7d7-v7m3",
       "creationTimestamp": "2019-11-19T10:26:52Z"
     },
     "timestamp": "2019-11-19T10:26:17Z",
     "window": "30s",
     "usage": {
       "cpu": "45602403n",
       "memory": "609968Ki"
     }
   }
 ]
}

EDIT:

You can also optionally access it from your pods since you created a Cluster Role Binding in your default Service Account with cluster-admin role.

As example, create a pod from a image that includes curl command:

$ kubectl run bb-$RANDOM --rm -i --image=ellerbrock/alpine-bash-curl-ssl --restart=Never --tty -- /bin/bash

Than you need to exec into you pod and run:

$ curl -k -X GET https://kubernetes.default/apis/metrics.k8s.io/v1beta1/nodes --header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" --insecure

Here we are passing the same TOKEN mentioned before in a complete different way.

I updated my answer including `type: LoadBalancer`. What cloud provider are you using? — Mark Watney, Nov 19 '19 at 12:54
I am using locally. But i can able to run api with NodePode service — UDIT JOSHI, Nov 19 '19 at 13:07
Is it possible to run metric server api in insecure way so that i can acess from inside pod as well with curl command , Because inside pod i can't generate token . Because kubectl command not running there — UDIT JOSHI, Nov 19 '19 at 13:09
I've updated my answer with instruction about how to access it from a pod. — Mark Watney, Nov 19 '19 at 14:00
If my answer helped you to achieve what you needed, please don't forget to accept my answer and/or upvote it. Thank you. — Mark Watney, Nov 20 '19 at 16:00
Your error is connect to code. Check is [this](https://stackoverflow.com/questions/9619030/resolving-javax-net-ssl-sslhandshakeexception-sun-security-validator-validatore) is connected. — Mark Watney, Nov 25 '19 at 08:06
I have one more issue, When I reboot the system . My metric server will be running but when i execute the api from inside the pod . it says "curl: (6) Couldn't resolve host 'kubernetes.default'" — UDIT JOSHI, Dec 02 '19 at 10:12
What cloud provider are you using? If you are using GKE, take a look at [this](https://stackoverflow.com/a/58976035/12153576). — Mark Watney, Dec 02 '19 at 10:25

How to expose kubernetes metric server api to curl from inside the pod?

1 Answers1