IPython notebook: How to connect to existing kernel?

Question

I am able to connect two ipython console session to one kernel by:

$ ipython console
In [1]: %connect_info  
{
   ... Content of JSON with info for connecting ...
}

Paste the above JSON into a file, and connect with:
    $> ipython <app> --existing <file>
or, if you are local, you can connect with just:
    $> ipython <app> --existing kernel-43204.json
or even just:
    $> ipython <app> --existing
    if this is the most recent IPython session you have started.

And accordingly I can substitute the <app> by console again

$ ipython console --existing kernel-43204.json

However, I want to share my kernel with ipython notebook so I can visualize my data. I tried and failed with:

$ ipython notebook --existing kernel-43204.json
[C 13:35:01.025 NotebookApp] Unrecognized flag: '--existing'

Any suggestion how I can work and switch between ipython console and ipython notebook?

Answer 1

There is no UI, nor API to do that with the notebook, there is an assumption for code simplicity that the notebook is the one that own and start the kernel. You will have to write your own KernelManager subclass and configure IPython to use it (+ write a bit of UI code, if you want it easy to use), for you to be able to select an already existing kernel.

Answer 2

I'll give you a solution the other way around. Instead of connecting a notebook to an existing kernel, you can easily connect an ipython session to a kernel that was started by a notebook.

1. Start your notebook. Now you have a running kernel.
2. In a code cell, run the magic command %qtconsole

Now you have a console and the notebook connected to the same kernel. You can run the magic command multiple times and have multiple consoles.

BTW, qtconsole is a very smart console. It is even better than the terminal one, especially if you are a Windows user.

Answer 3

Here’s the example of custom kernel manager that allows Jupyter notebook to kernel created externally.

https://github.com/ebanner/extipy

It’s hacky solution at best.

Jupyter folks can hopefully create such custome kernel class and include it in package and enable it via simple —existing switch. I don’t see any reason why they can’t do that.

Answer 4

None of the other answers here worked for our use case for various reasons, but after a fair amount of hacking we found the following solution. Some work is still needed to put this to use, but it should be enough to get started; just run each of the following blocks in a Jupyter notebook.

First, spool up a remote kernel for us to connect to:

import IPython
import inspect

test_var = "HELLO WORLD"

def embed():
    caller_frame = inspect.stack()[1][0]
    scope = {}
    scope.update(caller_frame.f_globals)
    scope.update(caller_frame.f_locals)

    IPython.embed_kernel(local_ns=scope)
embed()

Now we connect to the remote kernel using a BlockingKernelClient

import jupyter_client
connection_file = jupyter_client.find_connection_file("kernel-1337.json")

client = jupyter_client.BlockingKernelClient()
client.load_connection_file(connection_file)
client.start_channels()

At this point, we're able to send text to the remote kernel and have it interpreted as python. Note that test_var was never defined in our notebook's kernel, but we can still print it out like normal:

client.execute_interactive("print(test_var)")

Plotting works like normal too, even though we're sending data to the remote shell:

client.execute_interactive("import matplotlib.pyplot as plt; plt.scatter([1], [1])");

Now we just need to hide this hack from the user. We can do that using ipython input transformer hooks:

def parse_to_remote(lines):
    new_lines = []
    for line in lines:
        line_wrapped = f"client.execute_interactive({repr(line)});"
        new_lines.append(line_wrapped)
    return new_lines

ipy = get_ipython()
ipy.input_transformers_post.append(parse_to_remote)
del parse_to_remote

The del parse_to_remote is important; without it, the local kernel will crash.

After this point, any cells you run in your notebook will be parsed by the remote kernel instead of the local one, and outputs should just work like normal. If we print the remote-side test_var again, we can see that the code is running in the remote kernel (not the local, jupyter-spawned one):

print(test_var)

At this point, the Jupyter notebook will act as if it were connected directly to the remote kernel. Restarting the notebook will just reconnect to the remote kernel without restarting it.