9

I am serving jupyter notebook through a Kubernetes cluster. And I've set resources.limits to prevent someone from draining all of the host servers memory.

While one problem is that the jupyter notebook kernels after crash and automatic restart they do not throw any OOM errors after the container exceeds the memory, which will make the user very confused.

So how can I make the jupyter notebook raise the OOM error when running with Kubernetes?

aisensiy
  • 1,460
  • 3
  • 26
  • 42
  • Unfortunately the real question you are asking is unclear. Could you please clarify if you are looking for a solution inside of the Kubernetes? Do you seek for a solution which will show you a message that the pod containing Jupyter Notebook was OOM killed? Or you want to inform users somehow that it was killed because of memory limit? Or you are looking for a solution that will prevent the pod from being killed because of OOM. – aurelius Oct 01 '18 at 15:13
  • Thanks for your suggestion. I do wanna seek for a solution to show a message that the pod containing Jupyter Notebook was OOM killed. I already updated my question. – aisensiy Oct 02 '18 at 06:42
  • Correct me if I'm wrong, but your question should be - how I can inform user that there was a problem with corresponding kernel. – Michał Zaborowski Oct 12 '18 at 10:19

2 Answers2

4

If you have only one specific pod, you can monitor the events/logs, as in here:

kubectl get events --watch 
kubectl logs -f podname

That being said, not all events in a pod lifecycle are properly reported, as shown in kubernetes/kubernetes issue 38532 and the (abandoned) PR 45682.
But you should still see OOMKilled:true when docker inspect'ing the pod.

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • So I still can not make the python throw an OOM error? – aisensiy Oct 08 '18 at 07:05
  • @aisensiy python can throw an OOM, but you need to test if a docker inspect does report it as I mention in the last part of my answer. – VonC Oct 08 '18 at 07:09
0

Please look at solution from this post "Jupyter notebook: memory usage for each notebook" on killing a notebook when it exceeds certain amount of memory.

There is another nice interface (like top) from nbtop.

yoonghm
  • 4,198
  • 1
  • 32
  • 48