Reloading submodules in IPython

Question

Currently I am working on a python project that contains sub modules and uses numpy/scipy. Ipython is used as interactive console. Unfortunately I am not very happy with workflow that I am using right now, I would appreciate some advice.

In IPython, the framework is loaded by a simple import command. However, it is often necessary to change code in one of the submodules of the framework. At this point a model is already loaded and I use IPython to interact with it.

Now, the framework contains many modules that depend on each other, i.e. when the framework is initially loaded the main module is importing and configuring the submodules. The changes to the code are only executed if the module is reloaded using reload(main_mod.sub_mod). This is cumbersome as I need to reload all changed modules individually using the full path. It would be very convenient if reload(main_module) would also reload all sub modules, but without reloading numpy/scipy..

Would you care to elaborate more on `However, it is often necessary to change code in one of the submodules of the framework.` So why it's `necessary to change code`? Thanks — eat, Mar 19 '11 at 19:13
@eat: The framework is continually being developed, so there are constant changes to the code base. — Alain, Mar 19 '11 at 19:26

score 836 · Accepted Answer · edited Jan 25 '18 at 16:02

836

IPython comes with some automatic reloading magic:

%load_ext autoreload
%autoreload 2

It will reload all changed modules every time before executing a new line. The way this works is slightly different than dreload. Some caveats apply, type %autoreload? to see what can go wrong.

If you want to always enable this settings, modify your IPython configuration file ~/.ipython/profile_default/ipython_config.py[1] and appending:

c.InteractiveShellApp.extensions = ['autoreload']     
c.InteractiveShellApp.exec_lines = ['%autoreload 2']

Credit to @Kos via a comment below.

[1] If you don't have the file ~/.ipython/profile_default/ipython_config.py, you need to call ipython profile create first. Or the file may be located at $IPYTHONDIR.

edited Jan 25 '18 at 16:02

Brad Koch

19,267
19
110
137

answered Mar 22 '11 at 23:55

pv.

33,875
8
55
49

72

I have `c.InteractiveShellApp.extensions = ['autoreload']` and `c.InteractiveShellApp.exec_lines = ['%autoreload 2']` in my `~/.ipython/profile_default/ipython_config.py`. – Kos Jan 10 '13 at 12:24
4

That might be a performance hit, though, so use with caution. – asmeurer Jan 21 '13 at 07:10
3

The reload is only performed when you hit Enter in Ipython shell, and not usually noticeable. – pv. Feb 04 '13 at 23:32
It doesn't work with `from ... import *` type of import, right? Is there any way that works with that, too? – Baskaya Apr 12 '13 at 11:39
@Thorn: it works also for those to some degree, but does not track module dependencies. You can check for yourself to which degree this is true --- there are probably some corner cases that do not work as intended. – pv. Apr 12 '13 at 16:52
4

This works pretty fine when debugging packages, so what is the purpose of dreload, it seems dreload is too invasive and prints error when some packages like matplotlib are loaded. – dashesy Jun 02 '13 at 15:28
4

If you use @Kos method, make sure c is defined: c = get_config() – Tickon Apr 01 '15 at 12:19
1

Just remember that C extension modules (such as those from f2py) cannot be reloaded. This is in the [documentation](http://ipython.readthedocs.org/en/stable/config/extensions/autoreload.html), but I took me some time to notice it! – Ramon Crehuet Jan 18 '16 at 08:23
1

@Kos, what is the difference between the `load_ext autoreload` and `autoreload 2`, and `deepreload` modules in ipython? http://ipython.readthedocs.io/en/stable/api/generated/IPython.lib.deepreload.html?highlight=deepreload Does autoreload use `deepreload` in the background? – alpha_989 Apr 08 '18 at 21:44
Unfortunately this doesn't work on the "current" script. – CMCDragonkai May 08 '18 at 05:27
You shouldn't assign new values to `extensions` and `exec_lines`, you should append to the list so as not to override the existing extensions and lines assigned in those lists. My `ipython_config.py` contains `c.InteractiveShellApp.extensions.append('autoreload'); c.InteractiveShellApp.exec_lines.append('%autoreload 2') ` – Ziv Nov 14 '18 at 12:37
For Windows users that's C:\Users\\.ipython\profile_default\ipython_config.py – joseph Oct 07 '19 at 02:23
Alas, this kills state, even with a `try ...` trick https://stackoverflow.com/q/62204805/850781 – sds Jun 12 '20 at 16:54
FYI - if you're confused about get_config, ipython config getting started: https://ipython.org/ipython-doc/stable/config/intro.html – mVChr Mar 14 '22 at 17:47
This does not work for me. I execute the code stated here and things just do not seem to reload! – piedpiper May 14 '22 at 22:40
Just a note; this works when I run this in a cell at the *beginning* of a notebook (probably because I need to run this *before* importing any modules) – piedpiper Jul 03 '22 at 04:31

Jérôme Pouiller · Answer 2 · 2021-07-12T08:49:45.787

103

Module named importlib allow to access to import internals. Especially, it provide function importlib.reload():

import importlib
importlib.reload(my_module)

In contrary of %autoreload, importlib.reload() also reset global variables set in module. In most cases, it is what you want.

importlib is only available since Python 3.1. For older version, you have to use module imp.

I suggest to read documentation of importlib.reload() to get the list of all caveats of this function (recursive reload, cases where definitions of old objects remain, etc...).

edited Jul 12 '21 at 08:49

answered Jan 30 '19 at 12:03

Jérôme Pouiller

9,249
5
39
47

17

Over the past two years I've googled this question multiple times, and each time your answer is the perfect one I'm looking for. Thank you. – A Kareem Nov 03 '20 at 07:16
can only reload modules though. – bonobo Sep 07 '22 at 11:20
This works provided that if you want to reload a submodule named mymodule.data_class, then you have to write the code- importlib.reload(mymodule.data_class). – Prem Jun 28 '23 at 06:42

score 40 · Answer 3 · edited May 23 '17 at 10:31

40

In IPython 0.12 (and possibly earlier), you can use this:

%load_ext autoreload
%autoreload 2

This is essentially the same as the answer by pv., except that the extension has been renamed and is now loaded using %load_ext.

edited May 23 '17 at 10:31

Community

1
1

answered Feb 07 '12 at 16:27

RafG

1,374
12
16

1

For me in Jupyter lab this work and it is short enough to put into all notebooks who need it. – Nicola Mingotti Sep 10 '21 at 17:05

score 25 · Answer 4 · edited May 23 '17 at 11:54

25

For some reason, neither %autoreload, nor dreload seem to work for the situation when you import code from one notebook to another. Only plain Python reload works:

reload(module)

Based on [1].

edited May 23 '17 at 11:54

Community

1
1

answered Sep 28 '16 at 00:48

Dennis Golomazov

16,269
5
73
81

15

In Python 3.4+, you can find reload in the importlib module. See [this question](https://stackoverflow.com/a/27387813/525169). – Praveen Jun 04 '17 at 21:26
1

This method works when adding an instance method, unlike the `%autoreload` technique. There's an [open bug report to add support to `%autoreload` for this.](https://github.com/ipython/ipython/issues/8771). – ijoseph Jun 15 '18 at 20:34

Sven Marnach · Answer 5 · 2011-03-21T12:28:40.407

20

IPython offers dreload() to recursively reload all submodules. Personally, I prefer to use the %run() magic command (though it does not perform a deep reload, as pointed out by John Salvatier in the comments).

edited Mar 21 '11 at 12:28

answered Mar 20 '11 at 12:16

Sven Marnach

574,206
118
941
841

13

I think that (unfortunately) %run script.py only reloads the script you're calling, not the packages it imports. If you're trying to debug a package you're building, this can be a pain. – John Salvatier Mar 20 '11 at 21:29
5

NB. `dreload` has been replaced in recent IPython (e.g. IPython 6.0) by [deepreload](http://ipython.readthedocs.io/en/stable/api/generated/IPython.lib.deepreload.html). – dan mackinlay Apr 18 '18 at 03:05

score 7 · Answer 6 · answered Jun 06 '18 at 11:48

http://shawnleezx.github.io/blog/2015/08/03/some-notes-on-ipython-startup-script/

To avoid typing those magic function again and again, they could be put in the ipython startup script(Name it with .py suffix under .ipython/profile_default/startup. All python scripts under that folder will be loaded according to lexical order), which looks like the following:

from IPython import get_ipython
ipython = get_ipython()

ipython.magic("pylab")
ipython.magic("load_ext autoreload")
ipython.magic("autoreload 2")

This also seems to work if you're running your script with e.g. `%run script.py`from the IPython REPL — Patrick Sanan, Oct 03 '19 at 08:42

Y.H Wong · Answer 7 · 2011-03-19T22:27:03.150

5

How about this:

import inspect

# needs to be primed with an empty set for loaded
def recursively_reload_all_submodules(module, loaded=None):
    for name in dir(module):
        member = getattr(module, name)
        if inspect.ismodule(member) and member not in loaded:
            recursively_reload_all_submodules(member, loaded)
    loaded.add(module)
    reload(module)

import mymodule
recursively_reload_all_submodules(mymodule, set())

This should effectively reload the entire tree of modules and submodules you give it. You can also put this function in your .ipythonrc (I think) so it is loaded every time you start the interpreter.

edited Mar 19 '11 at 22:27

answered Mar 19 '11 at 22:13

Y.H Wong

7,151
3
33
35

That looks good, however, it might be that it is not covering modules or members of modules that are imported using `from ... import ...` or `import ... as`. At least that's giving me often some trouble when working interactively on the terminal. I've moved on to use stored macros in IPython that do the necessary imports and setups to start working in a predefined state. – Bernhard Mar 19 '11 at 22:39
It actually does cover `from ... import ...` and `import ... as` as long as the thing you imported is a module. The only thing that it doesn't cover is modules in a package that weren't loaded from it's `__init__.py` file. For packages, you can probably check if the module's `__path__` attribute is a directory. If it is, traverse it and recursively import all the modules you could find. I didn't write this part because the author has not asked for a solution for packages. – Y.H Wong Mar 19 '11 at 23:23
Indeed this looks good. I thought about this possibility, at the same I would expect that there would be some built in functionality, i.e. based on [this](http://ipython.scipy.org/ipython/ipython/ticket/154). However, it was not clear to me how to use this. After some digging that should have occurred before I posted the original question, I found this [extension](http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython). – Alain Mar 20 '11 at 00:27
Also you can use `pkgutil` to get all submodules in a package, even if the package does not import the submodules into the top module. http://stackoverflow.com/a/1707786/1243926 – Sergey Orshanskiy Jan 02 '15 at 18:45
you also have to do:for module in `sys.modules:` – aoeu256 Sep 12 '19 at 17:13

ryanjdillon · Answer 8 · 2018-12-18T08:50:04.090

4

My standard practice for reloading is to combine both methods following first opening of IPython:

from IPython.lib.deepreload import reload
%load_ext autoreload
%autoreload 2

Loading modules before doing this will cause them not to be reloaded, even with a manual reload(module_name). I still, very rarely, get inexplicable problems with class methods not reloading that I've not yet looked into.

edited Dec 18 '18 at 08:50

answered Nov 24 '16 at 12:14

ryanjdillon

17,658
9
85
110

score 3 · Answer 9 · answered Nov 23 '16 at 05:04

3

Another option:

$ cat << EOF > ~/.ipython/profile_default/startup/50-autoreload.ipy
%load_ext autoreload
%autoreload 2
EOF

Verified on ipython and ipython3 v5.1.0 on Ubuntu 14.04.

answered Nov 23 '16 at 05:04

Leo

1,710
2
26
29

score 3 · Answer 10 · answered Mar 25 '21 at 22:12

Prior to your module imports include these lines, where the first tests whether or not the autoreload extension has already been loaded:

if 'autoreload' not in get_ipython().extension_manager.loaded:
    %load_ext autoreload
%autoreload 2

import sys
    .
    .
    .

rdmolony · Answer 11 · 2020-07-29T21:14:48.543

I hate to add yet another answer to a long thread, but I found a solution that enables recursive reloading of submodules on %run() that others might find useful (I have anyway)

del the submodule you wish to reload on run from sys.modules in iPython:

In[1]: from sys import modules
In[2]: del modules["mymodule.mysubmodule"] # tab completion can be used like mymodule.<tab>!

Now your script will recursively reload this submodule:

In[3]: %run myscript.py

score 1 · Answer 12 · answered Nov 13 '19 at 09:35

1

Note that the above mentioned autoreload only works in IntelliJ if you manually save the changed file (e.g. using ctrl+s or cmd+s). It doesn't seem to work with auto-saving.

answered Nov 13 '19 at 09:35

Hubert Grzeskowiak

15,137
5
57
74

I confirm this is the case also for PyCharm. – tsando Feb 12 '20 at 17:23

thanks_in_advance · Answer 13 · 2017-10-26T22:35:02.833

On Jupyter Notebooks on Anaconda, doing this:

%load_ext autoreload
%autoreload 2

produced the message:

The autoreload extension is already loaded. To reload it, use: %reload_ext autoreload

It looks like it's preferable to do:

%reload_ext autoreload
%autoreload 2

Version information:

The version of the notebook server is 5.0.0 and is running on: Python 3.6.2 |Anaconda, Inc.| (default, Sep 20 2017, 13:35:58) [MSC v.1900 32 bit (Intel)]

score 0 · Answer 14 · answered Sep 12 '19 at 17:20

0

Any subobjects will not be reloaded by this, I believe you have to use IPython's deepreload for that.

answered Sep 12 '19 at 17:20

aoeu256

385
3
7

Reloading submodules in IPython

14 Answers14

Linked

Related