8

I want to switch my notebook easily between different kernels. One use case is to quickly test a piece of code in tensorflow 2, 2.2, 2.3, and there are many similar use cases. However I prefer to define my environments as dockers these days, rather than as different (conda) environments.

Now I know that you can start jupyter in a container, but that it not what I want. I would like to just click Kernel > use kernel > TF 2.2 (docker), and let jupyter connect to a kernel running in this container.

Is something like that around? I have used livy to connect to remote spark kernels via ssh, so it feels like this should be possible.

Roelant
  • 4,508
  • 1
  • 32
  • 62

1 Answers1

14

Full disclosure: I'm the author of Dockernel.

By using Dockernel

Put the following in a file called Dockerfile, in a separate directory.

FROM python:3.7-slim-buster

RUN pip install --upgrade pip ipython ipykernel
CMD python -m ipykernel_launcher -f $DOCKERNEL_CONNECTION_FILE

Then issue the following commands:

docker build --tag my-docker-image /path/to/the/dockerfile/dir
pip install dockernel
dockernel install my-docker-image

You should now see "my-docker-image" option when creating a new notebook in Jupyter.

Manually

It is possible to do this kind of thing without much additional implementation/tooling, it just requires a bit of manual work:

  1. Use the following Dockerfile:
FROM python:3.7-slim-buster

RUN pip install --upgrade pip ipython ipykernel

  1. Build the image using docker build --tag my-docker-image .

  2. Create a directory for your kernelspec, e.g. ~/.local/share/jupyter/kernels/docker_test (%APPDATA%\jupyter\kernels\docker_test on Windows)

  3. Put the following kernelspec into kernel.json file in the directory you created (Windows users might need to change argv a bit)

{
 "argv": [
  "/usr/bin/docker",
  "run",
  "--network=host",
  "-v",
  "{connection_file}:/connection-spec",
  "my-docker-image",
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "/connection-spec"
 ],
 "display_name": "docker-test",
 "language": "python"
}

Jupyter should now be able spin up a container using the docker image specified above.

Błażej Michalik
  • 4,474
  • 40
  • 55
  • It seems on Windows (running the linux docker) I run into this issue https://github.com/jupyter/notebook/issues/4998. Its just not entirely clear from the thread how I should fix it. Do you have this issue before? – Roelant Sep 03 '20 at 09:37
  • 1
    @Roelant it's because that file was somehow edited by the container, i.e. the kernel inside the container wrote there. Since it was binded to the container filesystem via docker daemon, which runs on admin privileges, the file got access permissions where only administrator can edit it. This is very setup specific. Does it occur at the first start of the kernel or after a retry? – Błażej Michalik Sep 03 '20 at 11:22
  • It occurs when I start the kernel for the first time. I can get a different error if I use the `ENV JUPYTER_RUNTIME_DIR=/tmp/runtime` and adjust the location on the `-f` flag `Timeout waiting for kernel_info reply`. But if I also adjust the `{connection_file}:/tmp/runtime/connection_spec` I again get the error with permissions. No matter if I run jupyter as admin or not. – Roelant Sep 03 '20 at 15:02
  • @Roelant you shouldn't really adjust these parameters this way. The right side (after colon) of `-v` must be equal to the argument passed as `-f`. Check whether binding any file to container changes its permissions. Use `docker run -it -v path\to\file:/test-file my-docker-image /bin/bash`, and check permissions on `path\to\file` after that. The file must exist beforehand. Also, if you get a directory inside the container in place of the `test-file`, something is wrong. – Błażej Michalik Sep 03 '20 at 18:22
  • Do you have any thoughts or pointers about how a similar thing can be achieved using `singularity`? HPC admins don't like to let docker run on the cluster as it requires elevated privilege. If it helps, singularity can natively run docker images. – axolotl Oct 25 '21 at 19:01
  • @axolotl no idea, I haven't used it. – Błażej Michalik Oct 26 '21 at 00:26