2

so - I love jupyter-lab ... it's great that you can work on a large complicated process step by step.

However - once you get it working, I find I always have to either

  • add and parse arguments
  • put it in a loop

because almost always I step through and get it working with one example, but I might then need to run it against a million instances- or make it a tool where you can say - point it at a database or directory or whatever.

And while you can export it as a python file, put it all in a function, then add your argument parsing or your loop - suddenly you've lost the notebook element of it - and you can't easily go back.

I'm just wondering if anyone's come out with some sort of technique to basically achieve both - ie have a large notebook split into steps, but then somehow run the the whole thing with different sets of arguments, possibly millions of times - ideally without losing jupyter-lab. ie - sort of like putting a for loop across the whole thing or have some sort of "go to cell" or something...

or just solving the underlying problem some completely different way I've never thought of.

Darren Oakey
  • 2,894
  • 3
  • 29
  • 55
  • I would write script which reads all code from notebook file and generate norma .py file. And in notebook file I would put some comments/docstrings with extra information which I would use in generator. ie. first cell coull have commend with function name and arguments - ie. `def main(filename):` - and then generator could put all code in function `def main(filename):`, etc. Maybe it could also use comments simiar to templates in `Flask`, `Django` to create loops - `{% for .... %}` ... cells with code .... `{% end for %}` – furas Jun 22 '21 at 06:12
  • using Google I found [Best practices for turning jupyter notebooks into python scripts](https://stackoverflow.com/questions/32183164/best-practices-for-turning-jupyter-notebooks-into-python-scripts/45244438) and one of answers has link to [NotebookScripter](https://github.com/breathe/NotebookScripter) which can load notebook and you can run it as function - so partially it can resolve problem because you can run it in loop for different values. – furas Jun 22 '21 at 06:33
  • 1
    Just to make sure: have you tried [jupytext](https://github.com/mwouts/jupytext) and [papermill](https://github.com/nteract/papermill)? – krassowski Jun 22 '21 at 06:54
  • will investigate all of these thanks – Darren Oakey Jun 22 '21 at 11:09

0 Answers0