After an OOM how can I detect the pod that caused it?

Question

I had some issues running some pods on a cluster, I want to know the way to detect which pod (and rc) is causing OOM on my nodes after the exception is thrown. I cannot access the node to check logs and kubectl describe node doesn't give me much information about this.

Thanks :)

score 3 · Accepted Answer · answered Jul 08 '16 at 15:34

3

Have you try running kubectl get events --watch to monitor the events on k8s and monitor the pod as well with kubectl logs -f podname

answered Jul 08 '16 at 15:34

PaulMB

457
1
4
17

3

Yes, with the events I could see the OOM but I didn't know what pod was consuming the memory. – Tim Givois Jul 11 '16 at 21:28

score 2 · Answer 2 · answered Jul 11 '16 at 21:28

The only way I found to track what is happening is waiting before an OOM is thrown on the web UI link and tracking the pod's memory used (I had a node with 1.75 GB and a pod that was consuming 1.3GB). I tried to see the memory allocation with kubectl describe node [nodename] but I couldn't.

After an OOM how can I detect the pod that caused it?

2 Answers2

Linked