1

I am trying to monitor filesystem usage for pods in k8s. I am using Kubernetes (microk8s) and hostpath persistent volumes. I am running Kafka along with a number of producers to see what happens when I go past the PVC size limit among other things. I have tried getting information from the API server but it is not reported there. Since it is only using hostpath, that kind of makes sense. It is not a dynamic volume system. Doing df on the host just shows all of the volumes with the same utilization as the root filesystem. This is the same result using exec -- df within the container. There are no pvcRefs on the containers using api server, which kind of explains why the dashboard doesn't have this information. Is this a dead end or does someone have a way around this limitation? I am now wondering if the PVC limits will be enforced.

user2065750
  • 58
  • 5
  • I think this kind of answers my question. Basically, the hostpath storage doesn't implement these metrics. There is a list on one of the answers with the ones that do. I still wonder if storage limits would be enforced but I plan on scaling this to more than one node (which hostpath won't do, at least not well) so I will need to replace it at some point. I guess it is moot. https://stackoverflow.com/questions/44718268/how-to-monitor-disk-usage-of-kubernetes-persistent-volumes – user2065750 Mar 22 '21 at 18:51

1 Answers1

1

Since with hostPath your data is stored directly on the worker you won't be able to monitor the usage. Using hostPath has many drawbacks and while its good for testing it should not be used for some prod system. Keeping the data directly on the node is dangerous and in the case of node failure/replacement you will loose it. Other disadvantages are:

  • Pods created from the same pod template may behave differently on different nodes because of different hostPath file/dir contents on those nodes

  • Files or directories created with HostPath on the host are only writable by root. Which means, you either need to run your container process as root or modify the file permissions on the host to be writable by non-root user, which may lead to security issues

  • hostPath volumes should not be used with Statefulsets.

As you already found out it would be good idea to move on from hostPath towards something else.

acid_fuji
  • 6,287
  • 7
  • 22
  • 1
    Thanks for taking the time to answer. I tried to use an operator on a micork8s cluster setup with the storage addon and it failed miserably. It doesn't account for the hostpath. Running my app deployments worked but it will only use one node for stateful containers (from what I see so far). Some of this will be on the edge so statefulness will be limited. I do see a number of container processes running as other users (grafana is 472 for example) but their pvcs are all root. – user2065750 Mar 23 '21 at 18:37