How can I use super() in both elegant and safe (regarding Python module reloading) way?

Question

I discovered an annoying behaviour of reload() (both Python 2 builtin and from importlib) that I am trying to walkarround.

I am analysing data in interactive Python interpreter. My code is organised in modules (both Python 2 and 3 compatible) which I often change.

Restarting the interpreter is not feasible due to long time of loading data, so I prefer to recursively reload modules instead.

The problem is that reload() updates the code but preserves the module global scope (it applies to Python 3 importlib.reload() as well). It seems to be harmful for methods using super() (it took me a while to realise what is going on).

The minimal failing example for a module bar.py:

class Bar(object):
    def __init__(self):
        super(Bar, self).__init__()

is:

>>> import bar
>>> class Foo(bar.Bar):
...     pass
... 
>>> reload(bar)
<module 'bar' from '[censored]/bar.py'>
>>> Foo()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "[censored]/bar.py", line 3, in __init__
    super(Bar, self).__init__()
TypeError: super(type, obj): obj must be an instance or subtype of type

I may:

use super() without arguments in Python 3 manner (which is not compatible with Python 2),
abandon it and call Bar.__init__(self) instead (which is harder to maintain and discouraged),
monkey-patch the class adding a class attribute containing a circular reference to the class itself.

None of the ideas I like. Is there any other way of dealing with the issue?

Your `bar.py` already fails because you put `super(Bar, self).__init__()` at class level instead of in an `__init__` method. — user2357112, Nov 03 '17 at 19:42
Your usage of `super` is invalid in fact. `super` is used within methods of a class not in the class body. — GIZ, Nov 03 '17 at 19:43
Also, reloading modules is just really hard to get right in general, and `reload` only handles the very simplest cases. Your best bet will always be to restart Python from scratch. — user2357112, Nov 03 '17 at 19:44
Don't retype it! Copy and paste from code you've actually run. Much safer. — user2357112, Nov 03 '17 at 19:46
Typically I use `super(self.__class__, self).__init__()` on Python 2, I'm not sure if it works on Python 3 — bphi, Nov 03 '17 at 19:49
@bphi: **Never do that**, because it breaks as soon as you subclass that class and `self.__class__` is the subclass. If it was that simple, `super` never would have required the type argument in the first place. — user2357112, Nov 03 '17 at 19:51
@bphi: user2357112 is right, I learned that the hard way. :/ — abukaj, Nov 03 '17 at 19:57

user2357112 · Accepted Answer · 2017-11-03T20:14:37.613

You can't, because it's basically impossible to do anything elegantly and safely where module reloading is concerned. That kind of thing is super tricky to get right.

The specific way that problem is manifesting here is that Foo's superclass is still the old Bar, but the old Bar refers to itself by name, and the Bar name now refers to the new Bar in the namespace where it's looking. The old Bar is trying to pass the new Bar to super.

I can give you some options. All of them are inelegant and unsafe and probably going to give you weird surprises eventually, but all that is also true about module reloading.

Probably the most easily-understood option is to rerun the Foo class definition to get a new Foo class descending from the new Bar:

reload(bar)
class Foo(bar.Bar):
    pass

New instances of Foo will then use the new Bar, and will not experience problems with super. Old instances will use the old Foo and the old Bar. They're not going to have problems with __init__, because that already ran, but they're likely to have other problems.

Another option you can take is to update Foo's superclass so Foo now descends from the new Bar:

reload(bar)

Foo.__bases__ = (bar.Bar,)

New and old instances of Foo will then use the new Bar. Depending on what you changed in Bar, the new class may not be compatible with the old instances, especially if you changed what data Bar keeps on its instances or how Bar.__init__ performs initialization.

In my real-life example modules reload their imports recursively, so I hope code differences would not be the case. — abukaj, Nov 03 '17 at 20:25

score 0 · Answer 2 · answered Nov 03 '17 at 20:35

0

Sound like what you really want it to do re-import the module. The following allow that (in both Python 2 and 3).

import bar
import sys

class Foo(bar.Bar):
    pass

# Manually reload module.
_saved_reference = sys.modules['bar']  # Needed for Python 2.
del sys.modules['bar']

import bar  # Re-import instead of reload().
Foo()  # No error.

answered Nov 03 '17 at 20:35

martineau

119,623
25
170
301

What's `_saved_reference` for? – user2357112 Nov 03 '17 at 20:41
This solution has its own weird surprises, of course. Old references to `bar` and `Bar` will still refer to the old module and the old class, so `Foo` will still be using the old class, and any other modules that had imported `bar` will still have the old module. Reimporting packages this way is also likely to break access to submodules. Without restarting Python, any option is going to have weird surprises. – user2357112 Nov 03 '17 at 20:51
@user2357112: `_saved_reference` is needed because of something Python 2 does. See the accepted answer to [this question](https://stackoverflow.com/questions/5365562/why-is-the-value-of-name-changing-after-assignment-to-sys-modules-name) of mine that discusses it. – martineau Nov 03 '17 at 21:01
Huh. I thought that was an effect of interpreter shutdown, not part of the module destructor. – user2357112 Nov 03 '17 at 21:07

How can I use super() in both elegant and safe (regarding Python module reloading) way?

2 Answers2