How do I fix Kubernetes NodeUnderDiskPressure errors?

Question

After creating a simple hello world deployment, my pod status shows as "PENDING". When I run kubectl describe pod on the pod, I get the following:

Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  14s (x6 over 29s)  default-scheduler  0/1 nodes are available: 1 NodeUnderDiskPressure.

If I check on my node health, I get:

Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  OutOfDisk        False   Fri, 27 Jul 2018 15:17:27 -0700   Fri, 27 Jul 2018 14:13:33 -0700   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure   False   Fri, 27 Jul 2018 15:17:27 -0700   Fri, 27 Jul 2018 14:13:33 -0700   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     True    Fri, 27 Jul 2018 15:17:27 -0700   Fri, 27 Jul 2018 14:13:43 -0700   KubeletHasDiskPressure       kubelet has disk pressure
  Ready            True    Fri, 27 Jul 2018 15:17:27 -0700   Fri, 27 Jul 2018 14:13:43 -0700   KubeletReady                 kubelet is posting ready status. AppArmor enabled

So it seems the issue is that "kubelet has disk pressure" but I can't really figure out what that means. I can't SSH into minikube and check on its disk space because I'm using VMWare Workstation with --vm-driver=none.

Answer 1

This is an old question but I just saw it and because it doesn't have an answer yet, I will write my answer.

I was facing this problem and my pods were getting evicted many times because of disk pressure and different commands such as df or du were not helpful.

With the help of the answer that I wrote here, I found out that the main problem is the log files of the pods and because K8s is not supporting log rotation they can grow to hundreds of Gigs.

There are different log rotation methods available but I currently I am searching for the best practice for K8s so I can't suggest any specific one, yet.

I hope this can be helpful.

Answer 2

Personally I couldn't solve the problem using kube commands because ...
It was said to be due to an antivirus (McAfee). Reinstalling the company-endorsed docker-desktop version solved the problem.

Answer 3

as simohe commented, here are the treshholds: https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#hard-eviction-thresholds

Here are the Linux commands to check this on your Node:

• memory.available<100Mi free -hm | awk 'NR==2{print $7}' #output has to be higher than 100Mi
• nodefs.available<10% df -h / | awk 'NR==2{print $5}' #output has to be lower than 90%
• imagefs.available<15%
  • containerd: df -h /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs | awk 'NR==2{print $5}' #output has to be lower than 85%
  • docker: df -h /var/lib/docker | awk 'NR==2{print $5}' #output has to be lower than 85%
• nodefs.inodesFree<5% (Linux nodes) df -i / | awk 'NR==2{print $5}' #output has to be lower than 95%

To get a overview to where all your storage went use: du / -d 1 -h 2> /dev/null | sort -hr

Answer 4

Had the similar issue.

My_error_log : Warning FailedScheduling 3m23s default-scheduler 0/3 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had taint {node-role.kubernetes.io/controlplane: true}, that the pod didn't tolerate, 1 node(s) had taint {node.kubernetes.io/disk-pressure: }, that the pod didn't tolerate

For me, the / partition was filled to 82%. Cleaning up some unwanted folders resolved the issue. Command used:-

1. ssh uname@IP_or_hostname (login to the worker node )
2. df -h (to check the disk usage)
3. rm -rf folder_name (delete the unwanted folder,you are forcefully deleting the file, so make sure you really want to delete it).

I hope this can save someone's time.

Answer 5

Community hinted you the comments above. Will try to consolidate it.

The kubelet maps one or more eviction signals to a corresponding node condition.

If a hard eviction threshold has been met, or a soft eviction threshold has been met independent of its associated grace period, the kubelet reports a condition that reflects the node is under pressure.

DiskPressure

Available disk space and inodes on either the node’s root filesystem or image filesystem has satisfied an eviction threshold

So the problem might be not enough disk space or filesystem has run out of inodes. You have to learn about the conditions of your environment and then apply them in your kubelet configuration.

You do not need to ssh into the minikube since you are running it inside of your host: --vm-driver=none -

option that runs the Kubernetes components on the host and not in a VM. Docker is required to use this driver but no hypervisor. If you use --vm-driver=none, be sure to specify a bridge network for docker. Otherwise it might change between network restarts, causing loss of connectivity to your cluster.

You might try to check if there are some issues related to the mentioned topics:

kubectl describe nodes

Look at df reports:

df -i
df -h

Some further reading so you can grasp the topic: Configure Out Of Resource Handling - section Node Conditions.