1

We are consuming kubelet /stats/summary endpoint.

We noticed that the metrics returned are not always present and might be missing in some scenarios.

In particular we are interested in Rootfs.UsedBytes that in missing in minikube but present in other environments.

Command to retrieve /stats/summary from kubelet, notice that the port can vary in different k8s flavours

token=$(k get secrets <service-account-token-with-enough-privileges> -o json \
| jq .data.token -r | base64 -d -)

k run curler --rm -i --restart=Never --image nginx -- \
curl -X GET https://<nodeIP>:10250/stats/summary --header "Authorization: Bearer $token" --insecure


"pods": [
{
...
   "containers": [
   {
   ...
      "rootfs": {
         ...
         "usedBytes": 36864,
         ...
       }
  • Why is that?
  • Is there a similar metric more reliable?
  • Can add anything in Minikube to enable that?

EDIT:

It is possible that the issue is related to --driver=docker option of minikube

Wytrzymały Wiktor
  • 11,492
  • 5
  • 29
  • 37
GalloCedrone
  • 4,869
  • 3
  • 25
  • 41
  • Which version of Kubernetes did you use and how did you set up the cluster? – kkopczak Dec 14 '21 at 09:50
  • The cluster is a classical minikube, run with "minikube start" on OSX over Docker. Version 1.22. :) – GalloCedrone Dec 14 '21 at 09:55
  • Sorry for long response. I've analyzed your problem, but there are some understatement. Could you elaborate this sentence `"We noticed that the metrics returned are not always present and might be missing in some scenarios."` - what scenarios? If there are no metrics, does it work OK - are the pods to the cluster working? – kkopczak Dec 27 '21 at 21:18
  • For example they are missing in minikube when the driver is Docker on OSx. The cluster, however, seems perfectly fine and a "normal" user would not notice any difference in its operations. – GalloCedrone Dec 28 '21 at 10:17
  • 1
    Changing the driver to "Hyperkit" fixes the issue and that specific metric is available again under kubelet's /stats/summary – GalloCedrone Dec 28 '21 at 10:18

1 Answers1

1

To clarify I am posing community wiki answer.

The problem here was resolved by changing driver to Hyperkit.

According to the documentation:

HyperKit is an open-source hypervisor for macOS hypervisor, optimized for lightweight virtual machines and container deployment.

There are two ways to install HyperKit (if you have installed Docker for Desktop, you don't need to do anything - you already have HyperKit):

brew install hyperkit

See also this reference.

kkopczak
  • 742
  • 2
  • 8