37

This is a total noob Kubernetes question. I have searched for this, but can't seem to find the exact answer. But that may just come down to not having a total understanding of Kubernetes. I have some pods deployed across three nodes, and my questions are simple.

  1. How do I check the total disk space on a node?
  2. How do I see how much of that space each pod is taking up?
chuckw87
  • 647
  • 1
  • 6
  • 14
  • What version of kubernetes are you running? And where did you deploy the kubernetes cluster? – Blokje5 Jan 11 '19 at 15:40
  • Sorry did not see the Azure tag. Do you want a full monitoring solution, or just the ability to check what you are currently using? And pods don't really use Disk space unless you bind a volume to them. – Blokje5 Jan 11 '19 at 15:49
  • I am running Kubernetes version 1.11.3. Yes, just the ability to check what I am using. Also, if pods don't use disk space, does that mean I can just deploy as many pods that are are allowed to be allocated to a node? Which would be 110 in this case. – chuckw87 Jan 11 '19 at 15:55
  • Disk space is not the only limitation. You will most likely run out of memory and cpu on a node way before you run into issues with the Disk Space. If you want to monitor the health of your cluster you need to monitor more then only disk space. You could start here: https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/ with research. I personally prefer deploying prometheus on kubernetes to monitor the health of the cluster. – Blokje5 Jan 11 '19 at 15:58
  • Great, Thanks for the info! – chuckw87 Jan 11 '19 at 16:21
  • `df -h` for node disk space – Ivan Aracki Jan 13 '19 at 00:24
  • You can add the answer if you solve the problem. It can help others who find this. – Charles Xu Jan 14 '19 at 02:43

3 Answers3

23

For calculating total disk space you can use

 kubectl describe nodes

from there you can grep ephemeral-storage which is the virtual disk size This partition is also shared and consumed by Pods via emptyDir volumes, container logs, image layers and container writable layers

If you are using Prometheus you can calculate with this formula

sum(node_filesystem_size_bytes)
UDIT JOSHI
  • 1,298
  • 12
  • 26
  • Thanks for the answer. I'm using Amazon EKS to run kubeflow, so when each pod writes something in their pod it will take up some of `ephemeral-storage`? Also as long as I save something within a pod it is writing to `ephemeral-storage`? – haneulkim Jun 22 '23 at 04:42
2

I'm assuming you're using AKS as that's what the question is tagged with.

The worker nodes are just standard VMs with a whole load of scripts to bootstrap the Kubernetes cluster. Disk space is very important as every image layer you download will be cached on the server and by default the hard drive space of these servers can be very small (30GB IIRC) unless tweaked at creation. The partitioning schema is also not particularly tuned for container delivery.

You can use OMS and the container monitoring solutions in Azure to get a great insight into your cluster health. https://learn.microsoft.com/en-us/azure/azure-monitor/insights/containers or as mentioned above - you can use prometheus / Grafana or just ssh in and df -h to see what's going on (although I wouldn't advocate ssh access nodes).

The disk space on the nodes is very different to PVs mounted by the containers.

With regards to the max number of pods per node - I think the default is 30 unless you built the cluster specifically with a higher number.

Ben
  • 46
  • 1
2

The easiest way it to check the metrics from the Azure portal:

a. You can navigate to the Azure portal, look for the Kubernetes cluster that is affected.

b. In the left panel clicks on Node pools and navigate to the node pool which is reporting high disk utilization.

c. Click on nodes and select the affected node, under the monitoring tab, click on Data Bytes.

d. This will load the metrics chart that will display the average Disk Space utilization on the node.

Mohit Ganorkar
  • 1,917
  • 2
  • 6
  • 11
Syed Irfan
  • 21
  • 1