Set maximum number of cores for Jupyter notebook

Question

I share a computer with a colleague. They are running parallelized calculations there and I need to run Jupyter. I may use only a few cores, not all of them.

However, every time I run a cell which uses numpy in Jupyter, it tries to use as many cores as possible. While a colleague's calculation is running, Python takes half of the cores.

I tried to set niceness of Jupyter process to 19, so that its Python child processes inherit the niceness value and do not try to use all cores, but it does not work.

Is there a way how to limit Jupyter and its Python children to use some maximum number of cores? I hope that there is a variable for this limit.

https://stackoverflow.com/questions/42954206/how-can-we-configure-the-cpu-and-memory-resources-for-jupyter-notebook — gold_cy, Jan 25 '19 at 14:02
Maybe starting the process with [taskset](https://linux.die.net/man/1/taskset)? — Cristian Ramon-Cortes, Jan 25 '19 at 14:15

score 2 · Answer 1 · answered Jan 25 '19 at 15:17

Jupyter/notebook doesn't have any resource managers like that built in. Mostly that's because all of that stuff ended up in Jupyterhub, which is like another layer on top of the Jupyter architecture that's meant for making Jupyter play nicely with others in a mutli-user environment. Which is pretty much where you're at.

Jupyterhub does offer a way to set a hard limit on the number of cores it will use. See here for details.

Set maximum number of cores for Jupyter notebook

1 Answers1