Connecting a jupyter notebook to an existing and running ipython kernel

Question

Short version: I have a software running its own jupyter kernel. I want to connect to this running kernel from an external source. This works fine with the --existing option of jupyter console but jupyter notebook doesn't have such a functionality.

My issue is more or less this one: How do I connect a jupyter notebook to an existing IPython kernel? but the package showed there isn't working properly anymore. I did some code changes to it to make it "work" but I am still running into issues.

Context: I have a software with an embedded python3 version. This includes a bunch of c++ extensions made available through this embedded python (in the form of extra built-in modules). These are only available through the embedded python3 started with the software and can be used to interact with the software in various ways (similar to the bpy package in Blender for example). It also includes a bunch of modules(understand numpy, pandas, matplotlib and many others etc...)

To facilitate dev, I am "exposing" this embedded python and all modules as a jupyter kernel. I can successfully connect to it externally using jupyter console --existing kernel-<id>.json and get access to all the modules and use them to interact with the software.

Issue: There is no --existing option for notebooks, or code/script equivalents.

The one solution I found (the one used by the module mentioned here: How do I connect a jupyter notebook to an existing IPython kernel?) consists in having a custom kernel manager class, that you can then set with jupyter notebook --NotebookApp.kernel_manager_class=extipy.ExternalIPythonKernelManager --Session.key='b""'. It works by overwriting start_kernel and as it says: "It spins up a new kernel through the call to super().start_kernel(...) but then turns its attention to the kernel which was started by an external python process. Kernel restarts will restart the useless kernel and leave the existing kernel alone."

This is the core file class ExternalIPythonKernelManager file, but due to, I imagine, jupyter changes, it fails now (super(ExternalIPythonKernelManager, self).start_kernel(**kwargs).result() fails as a couroutine doesn't have a .result()). Here is my version of start_kernel that "works":

    async def start_kernel(self, **kwargs):
        kernel_id = await super(ExternalIPythonKernelManager, self).start_kernel(**kwargs)
        if self._should_use_existing():
            self._attach_to_latest_kernel(kernel_id)
        return kernel_id

But running jupyter notebook --NotebookApp.kernel_manager_class=extipy.ExternalIPythonKernelManager --Session.key='b""' now gets me

[I 14:33:17.997 NotebookApp] Kernel started: cdaf10a1-6af9-4f09-9e6f-c6298a0bb16e, name: python3
[IPKernelApp] ERROR | No such comm target registered: jupyter.widget.control
[IPKernelApp] WARNING | No such comm: 8563966b-82ac-4f35-8f9b-a98fceaf5f64

and when I go to the notebook, my custom Ipython kernel isn't available, only the default "Python 3" kernel.

Questions:

Any idea what could be going wrong? Could it be an issue with me replacing the non working tornado specific code with async/await based code?
Is there any other approach to connecting to an existing kernel for jupyter notebook?

Connecting a jupyter notebook to an existing and running ipython kernel

0 Answers0