I have a Jupyter notebook server running inside a Docker container on some cluster and I run some scala code (with Spark) in a bunch of these notebooks. They show up externally at some URL (http://blah.com:11111/tree). I need to automate this by writing a python script that can run notebooks from anywhere: open URL, run it, shutdown and restart kernels if needed, etc. There is some mentioning on this here and there, but it would be good to have a complete example: open HTTP request to notebook server in Python, interact with it, close it. The nbconvert approach wouldn't work in this case (assume no access to Jupyter server). The jupyter kernel_gateway sounds interesting but I couldn't find a good example which shows how to open HTTP request to notebook server in Python, interact with it, close it. There are some interesting information jupyter-client messaging and RESTful API, but again I couldn't find an example that I'm looking for. Thank you.
One way to do this is to create a notebook and put the following code in it:
import nbformat, os
from nbconvert.preprocessors import ExecutePreprocessor
print os.getcwd()
with open('./my_notebook.ipynb') as f:
nb = nbformat.read(f, as_version=4)
ep = ExecutePreprocessor(timeout=30, kernel_name='python2')
ep.preprocess(nb, {'metadata': {'path': './'}})
with open('./my_notebook.ipynb', 'wt') as f:
nbformat.write(nb, f)
This code can be a base for further automation: reading my_notebook.ipynb as a json file, changing the code and cells in it, storing it back and re-running it again. All kinds of loops and branches and multi-processing can be developed based on this. However, this doesn't work for spylon-kernel and scala/Spark notebooks. When I do kernel_name='spylon-kernel' it gives me exception NoSuchKernel: No such kernel named spylon-kernel
I think the solution should come from jupyter-client, but I couldn't find a good example.
