0

Use case: I'd like to automatically schedule the execution of certain notebooks weekly. These are notebooks that take a long time to run, and I'd like to be able to continue running if it errors from when it left off, after changing the cause of the error. It would also be useful to just inspect the run from the browser when I need it.

So what I'm looking for is to start a notebook (and start the execution of all cells) on a running server (eg. on port 8888), with either Python code or using the CLI - without using a browser.

Is there any way to achieve this?

Milán Vásárhelyi
  • 853
  • 5
  • 9
  • does this answer your question? https://stackoverflow.com/questions/35471894/can-i-run-jupyter-notebook-cells-in-commandline – Z Li May 24 '22 at 14:04
  • No, it does not. I want to run the notebook in an existing server, and open it in the browser, the solutions explained there do not do this. – Milán Vásárhelyi May 24 '22 at 14:33
  • How do they not do that? You can use chron to schedule runs with `jupyter nbconvert` or `papermill` or [jupytext](https://jupytext.readthedocs.io/en/latest/using-cli.html) or [ploomber](https://github.com/ploomber/ploomber)? Those can either be done directly on a command line via ssh or within Jupyter also on your server. Either way, after they are run you can also open them in your browser assuming Jupyter is on your server. Or fetch them from wherever they are to where you are running Jupyter in your browser. – Wayne May 24 '22 at 17:45
  • These resources may also help you sort out exactly what you want: [here](https://discourse.jupyter.org/t/dashbords-and-scheduling-in-notebooks/3529/4), [here](https://filipstollar.medium.com/how-to-schedule-a-jupyter-notebook-to-run-every-day-week-or-month-ae3f992f3afc). By the way, I think I recall Papermill handles errors better than `jupyter convert`. – Wayne May 24 '22 at 17:48
  • `papermill` and `nbconvert` do not have the ability to reuse the already running server, so I cannot connect to the running kernel in the browser. I will check out the other two, but I assume they work similarly. – Milán Vásárhelyi May 25 '22 at 07:24
  • What I want is to start the `example.ipynb` from code, and then be able to go localhost:8888/notebooks/example.ipynb, see the run in real-time, and be able to interrupt and change something as needed. – Milán Vásárhelyi May 25 '22 at 07:37
  • A trivial way to do this would be with a headless Chrome and Selenium, but that just feels unnecessary. With that said, there may be no other easy option. – Milán Vásárhelyi May 25 '22 at 07:38
  • With papermill you can do what you say. You kick off `example.ipynb` with papermill. As stated [here](https://stackoverflow.com/a/62933693/8508004), papermill saves the results after it executes each cell. So you make a copy when you want to check & view the results of the copy in your browser. That way you can interrupt & change things in original. If you want more interactive access to the kernel backing the notebook, you can attach a console to a running kernel & query that to get closer to what you want. See bottom of https://stackoverflow.com/a/60094686/8508004 & ... – Wayne May 25 '22 at 19:11
  • see [here](https://stackoverflow.com/q/41316857/8508004) – Wayne May 25 '22 at 19:12
  • I'm aware of what papermill can do, and I'm currently using it. This is still suboptimal for me, as I'd like to attach the browser to the running kernel, which does not seem possible with it. – Milán Vásárhelyi May 26 '22 at 08:03

0 Answers0