86

Does anyone know if it is possible to run an IPython/Jupyter notebook non-interactively from the command line and have the resulting .ipynb file saved with the results of the run. If it isn't already possible, how hard would it be to implement with phantomJS, something to turn the kernel on and off, and something to turn the web server on and off?

To be more specific, let's assume I already have a notebook original.ipynb and I want to rerun all cells in that notebook and save the results in a new notebook new.ipynb, but do this with one single command on the command line without requiring interaction either in the browser or to close the kernel or web server, and assuming no kernel or web server is already running.

example command:

$ ipython notebook run original.ipynb --output=new.ipynb

Noah
  • 21,451
  • 8
  • 63
  • 71
davidshinn
  • 1,916
  • 1
  • 16
  • 17
  • Ploomber (https://github.com/ploomber/ploomber) is the answer! So you can use it to convert your notebooks into .py scripts, automate the output to be .ipynb or an HTML report. Under the hood we use nbconvert but add more functionality like incremental builds. You can parameterize all of your output notebooks (that ran) into a single location and even convert them into other formats. It also saves most of the refactoring time in the process users have when going into production. – Ido Michael Feb 25 '22 at 13:07

6 Answers6

76

Yes it is possible, and easy, it will (mostly) be in IPython core for 2.0, I would suggest looking at those examples for now.

[edit]

$ jupyter nbconvert --to notebook --execute original.ipynb --output=new.ipynb

It is now in Jupyter NbConvert. NbConvert comes with a bunch of Preprocessors that are disabled by default, two of them (ClearOutputPreprocessor and ExecutePreprocessor) are of interest. You can either enabled them in your (local|global) config file(s) via c.<PreprocessorName>.enabled=True (Uppercase that's python), or on the command line with --ExecutePreprocessor.enabled=True keep the rest of the command as usual.

The --ExecutePreprocessor.enabled=True has convenient --execute alias that can be used on recent version of NbConvert. It can be combine with --inplace if desired

For example, convert to html after running the notebook headless :

$ jupyter nbconvert --to=html --execute RunMe.ipynb

converting to PDF after stripping outputs

$ ipython nbconvert --to=pdf --ClearOutputPreprocessor.enabled=True RunMe.ipynb

This (of course) does work with non-python kernels by spawning a <insert-your-language-here> kernel, if you set --profile=<your fav profile>. The conversion can be really long as it needs to rerun the notebook. You can do notebook to notebook conversion with the --to=notebook option.

There are various other options (timeout, allow errors, ...) that might need to be set/unset depending on use case. See documentation and of course jupyter nbconvert --help, --help-all, or nbconvert online documentation for more information.

ThomasH
  • 127
  • 1
  • 8
Matt
  • 27,170
  • 6
  • 80
  • 74
  • 1
    Thank you for pointing me in the right direction. Since the exact code wasn't posted, I forked one of the gists and added output code so that the following will run a notebook from the commandline just like a desired ```$ ipnbdocttest.py original.ipynb new.ipynb``` https://gist.github.com/davidshinn/6110231 – davidshinn Jul 30 '13 at 05:00
  • Great! this functionality will be added in nbconvert 2.0 with a nicer API, don't hesitate to edit wiki to add a link to your script. – Matt Jul 30 '13 at 08:52
  • 1
    @Matt Did this make it into ipython 2? If so, how do I use it? – Simon Byrne May 11 '14 at 09:38
  • 3
    No not in IPython 2, on its way for IPython 3 – Matt May 11 '14 at 12:28
  • Such a feature would be really interesting to make the notebooks a tool to perform reproducible analysis (like embedding the command in a makefile, i.e. running the notebook from the command line and then converting it to markdown/rst). Looking forward to it! – mgalardini Oct 01 '14 at 16:02
  • Great! I couldn't find it in the master though; could you point out how to invoke this function? Thanks? – mgalardini Oct 14 '14 at 11:01
  • 1
    `ipython nbconvert --to=html --ExecutePreprocessor.enabled=True RunMe.ipynb`. Will update answer. – Matt Oct 14 '14 at 14:28
  • Matt, maybe it makes sense to update response with info that it works in ipython 3? E.g. it does not work properly in ipython 2.4 for me (but does in version 3) – Qrilka Feb 09 '15 at 08:12
  • 7
    It seems like this is now (Ipython 3.1.0): `ipython nbconvert --to=html --execute RunMe.ipynb` – patricksurry Apr 30 '15 at 16:14
  • 1
    An cautionary tale on an edge case: nbconvert seems to have some timeout functionality that will fail if a cell's contents happens to have code that takes more than 5-10 seconds to execute. `[NbConvertApp] ERROR | Timeout waiting for execute reply` `[NbConvertApp] ERROR | failed to run cell: Empty()` `[NbConvertApp] ERROR | !make all` – NoahR Oct 09 '15 at 21:06
  • 6
    @NoahR Try `--ExecutePreprocessor.timeout=-1` [ref](http://nbconvert.readthedocs.io/en/latest/execute_api.html#execution-arguments-traitlets) – Lenna May 14 '16 at 18:34
  • 5
    The latest jupyter (4.1.0) indeed allows to run notebooks non-interactively `jupyter nbconvert --to notebook --execute --allow-errors notebook.ipynb` maybe the answer could be edited – mgalardini Jun 23 '16 at 09:36
  • In my case, the matplotlib figures were missing. Had to add `%matplotlib inline` to get figures in the output – wander95 Jan 07 '19 at 22:58
21

Until this functionality becomes part of the core, I put together a little command-line app that does just what you want. It's called runipy and you can install it with pip install runipy. The source and readme are on github.

paul
  • 1,201
  • 12
  • 9
  • runipy does not seem to work so well for me: it does not update the `signature` field, and as a consequence (well, I think it is a consequence), none of my SVG is actually rendered: IPython prefers to display the plain `__repr__` textual rendering instead. – akim Oct 27 '14 at 08:36
2

Run and replace original .ipynb file:

jupyter nbconvert --ExecutePreprocessor.timeout=-1 --to notebook --inplace --execute original.ipynb
mkoistinen
  • 7,724
  • 3
  • 41
  • 56
Manualmsdos
  • 1,505
  • 3
  • 11
  • 22
1

To cover some features such as parallel workers, input parameters, e-mail sending or S3 input/output... you can install jupyter-runner

pip install jupyter-runner

Readme on github: https://github.com/omar-masmoudi/jupyter-runner

Omar
  • 101
  • 6
0

One more way is to use papermill, it has Command Line Interface

Usage example: (you need to specify output path for execution results to be stored)

papermill your_notebook.ipynb logs/yourlog.out.ipynb

You also can specify required params if you wish with -p flag for each param:

papermill your_notebook.ipynb logs/yourlog.out.ipynb -p env "prod" -p tests "e2e"

one more related to papermill reply - https://stackoverflow.com/a/55458141/2957102

Sysanin
  • 1,501
  • 20
  • 27
-5

You can just run the iPython-Notebook-server via command line:

ipython notebook --pylab inline

This will start the server in non-interactive mode and all output is printed below the code. You can then save the .ipynb-File which includes Code & Output.

ala
  • 224
  • 3
  • 16
  • Thanks for the response, however I was looking for a more specific solution that required no intervention in the web browser nor to close out the kernel and web server. I edited the question to be more specific. – davidshinn Jul 28 '13 at 12:30
  • In this case, I don't know how to proceed. Do you plan to do this on a regular basis? The iPython-Notebook is designed to be used interactively in a browser and saving specific ipynb-Files from the command line seems rather odd to me. – ala Jul 28 '13 at 15:02
  • Yes, I plan to do this very often, if it were available. It would be very useful in cases where data sources change often and I want to see diagnostic output from using pandas interleaved with the code it produced, exactly what ipython notebook does very well. However, if I have many of these types of processes to run a day, it would be better if they could be batch run and then I could look at the output later, as if log files were produced. The problem with standard logging is that it is hard to see the code and the output interleaved with one another. – davidshinn Jul 28 '13 at 15:39
  • 4
    This seems like an amazingly useful use case to me. You can take a template notebook, parameterize it a little, and then get a printable fully-executed solution all without touching a browser. – holdenweb Aug 08 '14 at 05:53