11

I am working on a setup where several developers, working on different projects, all execute their code on a remote machine, using Jupyter notebook.

Since every projects requires a different virtualenv what happens now is that every developer for every projects, sets up a project specific virtualenv, installs notebook to it, runs it on a different port and connects to the remote machine through that port.

Is there a way to have 1 Jupyter notebook running on the remote machine, but be able to choose which virtualenv to use as kernel?

My main consideration is being able to expose only one port on the remote machine, but be able to use different virtual python environment for running the notebooks

denfromufa
  • 5,610
  • 13
  • 81
  • 138
bluesummers
  • 11,365
  • 8
  • 72
  • 108
  • You can do that but it has to be done on server side, user's won't be free to add new environment. Does that work for you? – Tarun Lalwani Dec 03 '17 at 19:21
  • All the users, have their own `/home/` folder on the server which they have permissions to edit - I need them to be able to create virtualenv's in their own space, but that one jupyter server will serve all of those environments. Just to be clear, the virtualenvs and the jupyter servers are on the remote machine. – bluesummers Dec 04 '17 at 09:00
  • Try running this in a workbook `! workon my-env-name && pip install ipykernel && python -m ipykernel install --user --name=my-env-name`. Then you should get a option to change the kernel in `Kernel -> Change kernel` – Tarun Lalwani Dec 05 '17 at 05:15
  • Can you please elaborate? I get it that you use virtualenvwrapper, and you install a lib called ipykernel, but what does the last command do? Does the jupyter server has to be part of the environment where the ipykernel is intalled? – bluesummers Dec 05 '17 at 14:40
  • I have not tried the setup yet but I believe for this to work ipython should not be installed inside the virtualenv you switch to – Tarun Lalwani Dec 05 '17 at 14:47
  • Did you get a chance to test this? Let me know your feedback – Tarun Lalwani Dec 07 '17 at 09:05
  • No I haven't, I've read about it a bit, but I didn't quite get the use... I don't really understand the flags. I've actually bountied this because I don't have much time available to test things and I was looking for a deliberate answer... If nothing will come up, I'm simply going to wait for JupyterLab which goes on beta in the 31st – bluesummers Dec 07 '17 at 09:21
  • did you read this document? http://ipython.readthedocs.io/en/stable/install/kernel_install.html – denfromufa Dec 09 '17 at 03:22
  • you can see example of how this works in Azure Notebooks – denfromufa Dec 09 '17 at 03:23
  • also read this blog post of potential pitfalls when using multiple environments: http://jakevdp.github.io/blog/2017/12/05/installing-python-packages-from-jupyter/ – denfromufa Dec 09 '17 at 03:25

2 Answers2

9

I am working on a setup where several developers

If you have many dev working on a remote machine you must use JupyterHub, JupyterHub is made for that, and JupyterHub is the first step toward easing your pain; if you do not use JupyterHub, things will go wrong.

Once you have JupyterHub installed, your devs will be able to login with their credential wit exposing a single port, and will be able to start/stop notebook servers without sshing in.

Once this is done, you can investigate multiple venv.

In each environment you want to install ipykernel. It is the module that knows how to talk to the notebook. And in each environment you need to issue the python -m ipykernel install --user --name=my-env-name as said in the comments below your posts. This register each env with Jupyter, telling it "Hey I exist expose me to your users". You may also decide to install this that does part of this automatically for you, but have some caveats.

As other commenters have pointed out you likely want to read Jake's post, and if you have several users you should absolutely always almost without questions use JupyterHub.

Matt
  • 27,170
  • 6
  • 80
  • 74
  • On one environment, the jupyter itself must be installed, how does the command `python -m ipykernel install --user --name=my-env-name` knows the "main" environment where the jupyter is installed at? (in all this setup, if I understand correctly, only one environment has to have jupyter in it) – bluesummers Dec 09 '17 at 16:42
  • 1
    Yes only one env (or even system python) have Jupyter Notebook. `python -m ipykernel install --user --name=my-env-name` does not need to "know" the Jupyter environment. It just drop files on disk on the right place. Jupyter find these files when it needs to. You can see what files jupyter can find by issuing `$ jupyter kernelspec list`. – Matt Dec 10 '17 at 14:05
  • Is there any way to avoid the overhead and patching security maintenance of `ipykernel` for each venv? The pip install brings in ipython, pyzmq jedi, tornado, traitlets, jupyter-client, and lots more. It ends up being 76 MB unpacked, 9 times bigger than the 8.5 MB for a bare-bones venv. – nealmcb Jan 27 '21 at 17:58
  • Note: you don't even need to restart the jupyter notebook server process. For me it worked to just refresh the `.../tree/...`Jupyter browser tab, but oddly enough it didn't work to just hit the "Refresh notebook list" button with the circling arrows next to "New". – nealmcb Feb 01 '21 at 00:25
0

Is there a way to have 1 Jupyter notebook running on the remote machine, but be able to choose which virtualenv to use as kernel?

This is how I managed to use multiple kernels in the same Jupyter notebook instance

conda install nb_conda

nb_conda is a notebook extention that allows you to manage conda environments from your notebook. It also allows you to switch kernels directly from the Kernal menu.
I have noticed that the above command installs it with a few extras (nbpresent, nb_anacondacloud) which can be optionally disabled.

jupyter-nbextension disable nb_anacondacloud --py --sys-prefix
jupyter-serverextension disable nb_anacondacloud --py --sys-prefix

jupyter-nbextension disable nbpresent --py --sys-prefix
jupyter-serverextension disable nbpresent --py --sys-prefix

If you do not yet use conda you should consider it for your package management and virtualenv needs [source].

I believe that this system does not have many of the pitfalls mentioned in jakesvdp's post that @denfromufa mentions as the notebook extension nb_conda should be dealing with all the internals.

Screenshots

conda environment manager Conda tab in jupyter notebook allows you to manage your environments right from within your notebook.

Change Kernel
You can also select which kernel to run a notebook in by using the Change kernel option in Kernel menu

Harsh
  • 166
  • 8