29

I try to get Total and Free disk space on my Kubernetes VM so I can display % of taken space on it. I tried various metrics that included "filesystem" in name but none of these displayed correct total disk size. Which one should be used to do so?

Here is a list of metrics I tried

node_filesystem_size_bytes
node_filesystem_avail_bytes
node:node_filesystem_usage:
node:node_filesystem_avail:
node_filesystem_files
node_filesystem_files_free
node_filesystem_free_bytes
node_filesystem_readonly
Uliysess
  • 579
  • 1
  • 8
  • 19

3 Answers3

45

According to my Grafana dashboard, the following metrics work nicely for alerting for available space,

100 - ((node_filesystem_avail_bytes{mountpoint="/",fstype!="rootfs"} * 100) / node_filesystem_size_bytes{mountpoint="/",fstype!="rootfs"})

The formula gives out the percentage of available space on the pointed disk. Make sure you include the mountpoint and fstype within the metrics.

AYA
  • 917
  • 6
  • 7
  • Using your method I get incorrect total disk size - I know I have 40Gb total but displayed is 42Gb. Can you explain why it might be like that? I just copy-pasted `node_filesystem_size_bytes{mountpoint="/",fstype!="rootfs"}` – Uliysess Aug 06 '19 at 08:21
  • Are you converting by power of 10? From byte to kb for example, division by 1024 is expected, which makes a difference when going from bytes to Gb. – AYA Aug 06 '19 at 08:45
  • I specified `node_filesystem_size_bytes{mountpoint="/",fstype!="rootfs"}` query with bytes unit – Uliysess Aug 06 '19 at 08:51
  • So, for example what is the output? – AYA Aug 06 '19 at 08:55
  • `42.0 GB` for query given above. For following query `node_filesystem_size_bytes{mountpoint="/",fstype!="rootfs"}*(1e+9)` with unit hardset to Gb it displays `42 GB` – Uliysess Aug 06 '19 at 14:04
  • In bytes, that should be a large number. What is the output it displays? – AYA Aug 06 '19 at 16:05
  • With `node_filesystem_size_bytes{mountpoint="/",fstype!="rootfs"}` and no unit type set - `42004086784` - here ofc displays bytes with no calculations; bytes - `42.0 GB`; Gb - `42 GB` – Uliysess Aug 07 '19 at 05:54
  • 1
    Ok, let’s divide the 42 trillion number with 1024 to arrive at KB and then again division by 1024 for Mb. At this point I calculate 40058 mb. Now divide by 1024 and it is ~39 GB. You can trust the output of this metrics and converting it to a percentage is generally useful. – AYA Aug 07 '19 at 06:47
  • Thank you for clear noobish explanation. I'll give it a day or two and will accept your answer as it's fairly close to fully correct answer (that extra 2Gb or 1Gb less might be harmful in a long run). Are you able to explain why there is a difference between calculating it by myself and calculating it automatically by Grafana/Prometheus with unit types specified? – Uliysess Aug 07 '19 at 07:04
  • Where do you see an output as GB? In my Prometheus console, only results in bytes units are displayed. – AYA Aug 07 '19 at 08:40
  • In Grafana Dashboard I added Singlestat panel. Selected Prometheus as Data Source that I created previously and when I select Unit in Options I get a result with calculated units. In Prometheus console I also see output only in bytes – Uliysess Aug 07 '19 at 08:58
  • My bad, my explanation lacked diff between GB and GiB, and my notation was wrong. So under Grafana - Unit - Disk(Metrics) outputs Gigabytes whereas Unit - Disk(IEC) outputs Gibibytes, what I was calculating. Df command outputs in GiB. units. – AYA Aug 08 '19 at 05:59
  • @AYA Thanks for the answer! If I understand correctly you filtered to mountpoint / (the root of the filesystem) because that's where the harddisk will be mounted on. That way you avoid picking up mountpoints like /run, which is just a tmpfs (memory-backed filesystem). You make sure the fstype is not rootfs as that is also a memory-backed filesystem. Is that correct? It would probably be good to add a explanation to your answer. – Almenon May 11 '21 at 01:19
12

FS utilization can be calculated as

100 - (100 * ((node_filesystem_avail_bytes{mountpoint="/",fstype!="rootfs"} )  / (node_filesystem_size_bytes{mountpoint="/",fstype!="rootfs"}) ))
David Buck
  • 3,752
  • 35
  • 31
  • 35
Krishnan Subramanian
  • 121
  • 1
  • 3
-1

According kubernetes-mixin you can use node:node_filesystem_usage, but if this metrics won't show your correct vales please ensure that device is set correctly.

I highly recommend using Prometheus Graph or Graphana Explore tab to check all available metrics.

FL3SH
  • 2,996
  • 1
  • 17
  • 25
  • How can I make sure device is set correctly? Using Panel "Graph" and calling `node:node_filesystem_usage` I receive only 1 line with tiny values. I see device but I dunno how I can verify if it's correct or eighter how to change it if it's wrong – Uliysess Aug 05 '19 at 12:02
  • As you can see in [node:node_filesystem_usage](https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/rules/rules.libsonnet#L382) deffinition it takes `node_filesystem_avail_bytes` and `node_filesystem_size_bytes`, so first check those metrics. – FL3SH Aug 05 '19 at 12:15