5

I'm looking into these two related questions: here and here.

I am seeing a behavior I do not expect in Python 3.6, which differs from behavior using plain reload in Python 2.7 (and 3.4). Namely, it seems that a module attribute that would be populated during module initialization or when re-exec-ing the module during a reload, is not restored after its local name is removed with del ... see below:

For Python 3.6:

In [1]: import importlib

In [2]: import math

In [3]: del math.cos

In [4]: math.cos
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-05b06e378197> in <module>()
----> 1 math.cos

AttributeError: module 'math' has no attribute 'cos'

In [5]: math = importlib.reload(math)

In [6]: math.cos
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-6-05b06e378197> in <module>()
----> 1 math.cos

AttributeError: module 'math' has no attribute 'cos'

In [7]: importlib.reload(math)
Out[7]: <module 'math' from '/home/ely/anaconda/envs/py36-keras/lib/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so'>

In [8]: math.cos
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-05b06e378197> in <module>()
----> 1 math.cos

AttributeError: module 'math' has no attribute 'cos'

For Python 2.7 (and Python 3.4):

In [1]: import math

In [2]: del math.cos

In [3]: math.cos
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-05b06e378197> in <module>()
----> 1 math.cos

AttributeError: 'module' object has no attribute 'cos'

In [4]: reload(math)
Out[4]: <module 'math' from '/home/ely/anaconda/lib/python2.7/lib-dynload/math.so'>

In [5]: math.cos
Out[5]: <function math.cos>

I have tried chasing the details of importlib from the source code down to the C-level module exec function, and I can't see any logic that would cause it to fail to write the re-initialized cos attribute back into the module's dict of module-scope globals.

My suspicion is that it's some type of bug in the C-level re-execution logic that looks at the attribute names found in the module's dictionary (the one that exists from whenever it was previously imported, and may be mutated to have deleted an attribute, like in my example), and then when using exec to write the module's execution side-effects into that dictionary, it's skipping key names (like cos) that don't exist in the module's namespace, which is different from the Python 2.7 behavior.

ely
  • 74,674
  • 34
  • 147
  • 228
  • From doc: When a module is reloaded, its dictionary (containing the module’s global variables) is retained. Read more https://docs.python.org/3/library/importlib.html#importlib.reload – Mazdak Feb 15 '18 at 18:01
  • 1
    @Kasramvd This does not explain it though. The dictionary is retained sure, but why are not new keys (like `cos`) added into the dictionary when the module is re-executed? The detail that it re-uses the existing module dictionary doesn't seem to address the question unfortunately, because it does that in Python 2.7 too, just that it re-populates it with module attributes as needed. – ely Feb 15 '18 at 18:06
  • note that the second version also works in python 3.4. So the change must have appeared in between – Jean-François Fabre Feb 15 '18 at 18:17
  • ["It is generally not very useful to reload built-in or dynamically loaded modules. Reloading `sys`, `__main__`, `builtins` and other key modules is not recommended. In many cases extension modules are not designed to be initialized more than once, and may fail in arbitrary ways when reloaded."](https://docs.python.org/3/library/importlib.html#importlib.reload) – user2357112 Feb 15 '18 at 18:24
  • 1
    @user2357112 But `math` clearly is designed to be reloaded, since it works in Python 2.7 and Python 3.4. I don't believe that comment from the docs is applicable to this problem either. – ely Feb 15 '18 at 18:25
  • "It used to work, so it must have been designed to work" is not a safe line of reasoning. – user2357112 Feb 15 '18 at 18:27

1 Answers1

4

I believe this is an (intended? unintended?) effect of PEP 489, an overhaul of extension module initialization. The PEP includes the following section:

Module Reloading

Reloading an extension module using importlib.reload() will continue to have no effect, except re-setting import-related attributes.

Due to limitations in shared library loading (both dlopen on POSIX and LoadModuleEx on Windows), it is not generally possible to load a modified library after it has changed on disk.

Use cases for reloading other than trying out a new version of the module are too rare to require all module authors to keep reloading in mind. If reload-like functionality is needed, authors can export a dedicated function for it.

The code change that appears to be responsible for this behavior was introduced in the commit that implemented PEP 489.

Even Python 3.4 didn't support truly reloading an extension module from a changed file; the closest it had was code to save a copy of the module's dict after initialization and copy the contents back into the module's actual dict on a reload. That code still exists, but it is no longer triggered for reloads, and I don't know if it was ever intended to be triggered for reloads. I believe that code is currently only used for subinterpreters.

Community
  • 1
  • 1
user2357112
  • 260,549
  • 28
  • 431
  • 505
  • 1
    I guess there isn't any satisfying answer beyond, "this is just the random thing that was committed for Python 3.6, since the devs consider this to essentially be undefined behavior." Thanks for linking the code change. I do think we should be careful about your line, "Python 3.4 didn't support truly reloading an extension module from a changed file..." The question is *not* about re-loading a dynamic extension when its source changed, rather it is just a question of why re-initialization is not *always* attempted, especially when the underlying module *did not* change. – ely Feb 16 '18 at 13:27