7

I have a pet project which I started as a Jupyter notebook. So far, I put all the Python code in the notebook.

At the start everything was fine. But over time the code I wrote in the notebook became more and more complex. Now it is close to unmanagenable: When I find an error, I need

  1. to navigate to the code part with the error (usually at the beginning of the notebook),
  2. fix the error there,
  3. go to (usually) the bottom of the notebook to trigger the execution of the code I changed.

I want to separate the code in two parts:

  1. One that will be stored as Python files and which I will edit using an editor (and/or an IDE).
  2. The code in the Jupyter notebook that calls code parts 1 and presents its output (i. e. use the Jupyter notebook as a user interface for the Python code from step 1).

Let's assume that the notebook runs on my local machine (Windows 7; Jupyter runs in Anaconda) and the Python files are also stored locally.

What are good ways to use code from IPython files such that I can modify this code frequently and fast?

By "frequently and fast" I mean "with as little steps as possible that are necessary to propagate changes from Python files to the notebook". The ideal solution would be something where I change one of the Python files, run one command, and then the changes are available in the Jupyter notebook. Or, to use an older analogy, I want it to be like PHP -- you change the code often and immediately see the results of your changes.

Update 1: I tried to use the solution with %load TestClass.py in a cell.

The problem is that the cell contents is not updated, if the file changes.

Example:

Let's say I put the text 

class TestClass:
    def __init__(self):
        print("TestClass constructor")

into TestClass.py. Then I create a cell in Jupyter notebook with %load TestClass.py. When I execute that cell, the code from TestClass.py is imported and the line %load TestClass.py gets commented out.

Now I change TestClass.py to

class TestClass:
    def __init__(self):
        print("TestClass constructor")
        print("change")

When I execute the cell, its contents has not changed.

Glory to Russia
  • 17,289
  • 56
  • 182
  • 325
  • Could this be achieved with magic cells with a main `.py` file and then importing the file using `%load` as in this post https://stackoverflow.com/questions/21034373/how-to-load-edit-run-save-text-files-py-into-an-ipython-notebook-cell, or by using `%%writefile myfile.py` to write all of the cells of the notebook to a python file which can be used for debugging? – Alessi 42 Jun 06 '19 at 15:58
  • @Alessi42 Thanks. This solution does not work because when the file is changed, the changes are not propagated to IPython (see update 1). – Glory to Russia Jun 08 '19 at 09:28
  • What happens if you just write a class with methods and import them in a main.ipynb? Example: Say you want to have a class for all visualizations, then you create a visualization.py. Then to use it in a main.ipynb wrapper. Simply call: `import visualization as vis vis.visualize_barchart(X)` And to manage both together you could use Pycharm IDE may be – coldy Jun 08 '19 at 20:53
  • @coldy What should I write in `main.ipynb`? – Glory to Russia Jun 09 '19 at 13:03

4 Answers4

12

Sounds like the autoload extension of IPython is exactly what you need. Simply plug

%load_ext autoreload
%autoreload 2

in one of the first cells of your Jupyter notebook and imported python modules are automatically reloaded on change. You also can do changes to installed python packages, provided you have installed them editable.

Tobias Windisch
  • 984
  • 5
  • 16
0

I've been working on a similar sort of problem. If you use Jupyterlab instead of the older Jupyter Notebooks, you can open multiple nb's at the same time, edit the one holding your functions, and then restart the kernel in the other to update when you import. Unfortunately you lose all your variables and need to run the entire notebook again but it allows the changes in the first nb to propogate into the second.

This isn't a perfect answer but has allowed me to keep working on my projects.

mneedham
  • 45
  • 4
0

You're writing:

and the line %load TestClass.py gets commented out.

I understand, that you comment it out. Why? Re-run the import and the class gets updated.

BTW !run TestClass.py should do the same - at least in Jupyter NB.

y4cine
  • 396
  • 2
  • 20
  • Re *you comment it out*: I didn't. It was commented out automatically after running the cell the first time. – Glory to Russia Jun 11 '19 at 06:17
  • You're right - I didn't know the behaviour of "%load". So I did the same with "%run" and the cell didn't get commented out. So for your future work, that's the command to use. Place it at the beginning of your notebook AND at any convenient place from where you want the reloading to happen. Just remember, that commands depending on the import will not be updated automatically (eg instances of classes, etc.). So either you re-run the notebook from the beginning to the current place, or you put the relevant code in a function that can be triggered from any place in the NB. – y4cine Jun 11 '19 at 12:14
0

I usually unload the modules I want to refresh then reimport. Objects bound to old code don't get refreshed and every now and then I have to restart the kernel but it's a lot faster than restarting the kernel each time. I would only suggest this for someone who already understands or is willing to understand the impact of removing modules from sys.modules. But it saves a lot of time for me.

See UnloadModules here.

My typical usage in Jupyter:

from coppertop import Unload
Unload("fred.joe")
from fred.joe.sally import arthur
arthur.doSomethingInteresting()
DangerMouse
  • 704
  • 7
  • 20