1

I have a website that performs file format conversions, and this service can be used by other websites via the window.postMessage system. It would be great if one could do this also via a Jupyter notebook. I run however into a serious issue. I can get python to create a Javascript command that sends a file in format A to the website, and I can access the response (file in format B) in Javascript after some delay, but I cannot get the response from Javascript back into Python for further processing. I have made an as-simple-as-possible notebook to demonstrate the issue. The reason seems to be that Python first executes all cells (in the Run all scenario) and only then looks at any new execution requests.

There are similar questions on stackoverflow, for example here: IPython Notebook Javascript: retrieve content from JavaScript variables. From what I found so far there is no solution to the problem. I find that very disappointing, because it would be so cool to create webservices that are directly accessible from Jupyter notebooks, without having to go the node.js route.

import asyncio, time
from IPython.display import display, Javascript

The code below inserts Javascript into the page: The 'sendRequest' function is just a five second delay, but it is intended to communicate with another webpage via window.postMessage.

futureResponse is an asyncio Future (python). It gets a result when sendRequest (Javascript) finishes and uses kernel.execute (Javascript) to call the responseHandler (python).

global futureResponse

def responseHandler(response):
  futureResponse.set_result(response)

display(Javascript("""
  window.responseHandler = (response) => {
    var kernel = IPython.notebook.kernel;
    var pyCommand = 'responseHandler(\\''+response+'\\')';
    kernel.execute(pyCommand);
    console.log('responseHandler called')
  };
  window.sendRequest = () => {
    return new Promise( (resolve,reject) => { setTimeout(()=>resolve('RESPONSE'),5000) } )
  }
"""))

The code below inserts Javascript that calls sendRequest. That will call responseHandler (Javascript) after the setTimeout delay (5 s), which calls kernel.execute which invokes responseHandler (python), which sets the result of futureResponse. However, if I use "await futureResponse", it never fulfills.

# the value of futureResponse will be set by the responseHandler
futureResponse = asyncio.Future()

display(Javascript("""
window.sendRequest().then(
  (response) => window.responseHandler(response)
)
"""))

# This does not work (futureResponse unresolved):
time.sleep(6)

# With await, the notebook keeps waiting forever.
#await futureResponse

print(futureResponse)

If the cell below is evaluated immediately after the previous cell (as in a Run all scenario), then futureResponse is still unresolved. But if it is evaluated 5 seconds later, futureResponse has the value "RESPONSE".

futureResponse

erpuntbakker
  • 91
  • 1
  • 4
  • If you're using asyncio, you shouldn't call `time.sleep(6)`, you should `await asyncio.sleep(6)` (or simply `await futureResponse`). Have you checked that `kernel.execute()` is asyncio-aware, i.e. that it runs the callback in the asyncio thread? If not, you should use `loop.call_soon_threadsafe(futureResponse.set_result, response)` to ensure that the event loop is woken up when the result is set. (Asyncio futures are meant to only be used from the thread that runs the asyncio event loop.) – user4815162342 Nov 16 '19 at 08:05
  • 1
    Thanks for the ideas; I implemented them but to no avail. I think the problem is that the execution of code-cells and additional requests that these cells generate is done via a first in, first out queue. If I uncommend the "await futureResponse" in the above code there is a catch-22: futureResponse gets resolved after all cells have run, but the cells are waiting for futureResponse to resolve. I have looked at the Jupyter notebook messaging system, and found that there are other messaging channels (shell, iopub, control, comm, input) to play with. This I tried to exploit in the answer below. – erpuntbakker Nov 17 '19 at 13:11

1 Answers1

3

I looked further into this and found a solution, but it really is a hack that might break with future versions of Jupyter notebook. I hope someone has a more future-proof answer.

The idea is to avoid the catch-22 that the code example creates by using the stdin channel of the Jupyter messaging system (https://jupyter-client.readthedocs.io/en/stable/messaging.html).

The code works as follows: when a code cell contains an input('prompt-text') command, this is sent to the notebook client over a dedicated stdin channel, which is independent of the normal cell execution stream. So I 'hacked' the code of the notebook client to capture input requests that are sent by the kernel, and provide an answer without prompting the user.

display(Javascript("""
  const CodeCell = window.IPython.CodeCell;

  CodeCell.prototype.native_handle_input_request = CodeCell.prototype.native_handle_input_request || CodeCell.prototype._handle_input_request
  CodeCell.prototype._handle_input_request = function(msg) {
    try {
      // only apply the hack if the command is valid JSON
      const command = JSON.parse(msg.content.prompt);
      const kernel = IPython.notebook.kernel;
      // in the future: send the command to a server and wait for a response.
      // for now: specify a 5 second delay and return 'RESPONSE'
      setTimeout(() => { kernel.send_input_reply('RESPONSE') },5000)
    } catch(err) {
      console.log('Not a command',msg,err);
      this.native_handle_input_request(msg);
    }
  }
"""))

response = input(json.dumps({"do":"something"}))
print(response)

This blocks execution for 5 seconds, then prints 'RESPONSE' and continues.

erpuntbakker
  • 91
  • 1
  • 4