4

My service running in a pod output too much log and cause low ephemeral storage. As a result, the pod is evicted and other services can't deploy to k8s.

So how I can determine pod resource ephemeral storage requests and limit to avoid this situation? I can't find any best practice about ephemeral storage.

Ruoxuan.Wang
  • 179
  • 1
  • 2
  • 13

3 Answers3

7

Note that by default, if you have not set any limits on ephemeral-storage the pod has access to the entire disk of the node it is running on, so if you are certain that the pod is being evicted because of this, then you are certain that the pod consumed it all. You can check this from kubelet logs, as kubelet is the guy in charge of detecting this behavior and evicting the pod.

Now, here you have two options. Either you can set an ephemeral-storage limit, and make a controlled pod eviction, or just get an external volume, map it into the container, and get the logs outside of the node.

You can also monitor the disk usage, as suggesting shubham_asati, but if it is eating it all, it is eating it all. You are just going to look at how it is getting filled out.

suren
  • 7,817
  • 1
  • 30
  • 51
  • Thanks, I found the root cause. Because my Flentd pod failed down. So it can't collect and rotate logs. As a result, the log file consumes the whole disk and cause pod evicted. – Ruoxuan.Wang Nov 27 '19 at 08:36
  • Is it recommended to set the ephemeral-storage limit and request? – Ruoxuan.Wang Nov 27 '19 at 08:38
  • Generally speaking, it is recommended to set all resource requests, yes. In this case you should also put the limits, so the termination of the pod will be done in more controlled way, as kubelet doesn't evict the pod that causes this problem, but evicts the one that will give more room in the node. In your situation (and in most situations) happens to be the same pod that causes the issue. – suren Nov 27 '19 at 09:57
2

I guess ephemeral storage for a pod can be defined as cpu request/limit. See this https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#local-ephemeral-storage but this feature is in beta stage K8's version 1.16.

To check namespace level resource consumption view https://kubernetes.io/docs/concepts/policy/resource-quotas/#storage-resource-quota.

You can set request/limit ephemeral storage for each pod .

Regarding your issue

  1. check namespace quotas for ephemeral storage using kubectl describe namespace
  2. try du -sh / inside a container.

then compare the storages from both outputs.

shubham_asati
  • 623
  • 5
  • 14
1

You need to deploy prometheus and grafana to find out how much memory and cpu are getting consumed by the pod. Then accordingly set those request and limits on that pod

Requests and limits setting for ephemeral storage is a new feature and is still in beta.You might have to wait few more months to use that feature.

However, if you are on k8s 1.18 then you can test Requests and limits setting for ephemeral storage

P Ekambaram
  • 15,499
  • 7
  • 34
  • 59