2

How does module reload work in python 3.7.1?

Setup and Summary

I am running python 3.7.1 in Linux. I am developing a C module, and it would be very handy to reload the module once it changed. I followed How do I unload (reload) a Python module?, but I cannot get it to work in this environment.

For a demo of my problem, I wrote a simple module based on the SPAM example in the tutorial that returns the build time of the module. It will never get reloaded.

Code

The implementation is the spam example from the tutorial. It has a single function hello which returns the build time:

return Py_BuildValue("s", __TIME__);

I am compiling and loading with the following python script:

import os
import helloworld
print(helloworld.hello('test'))
os.system("touch helloworld.c")
os.system("python3 setup.py build")
os.system("python3 setup.py install --user")
from importlib import reload
helloworld=reload(helloworld)
print(helloworld.hello('test'))

The module is imported, the main file is touched, it is compiled and installed, and then reloaded.

Output

The module should show the new compilation time after reload, but the output does not change (I am leaving out some debug messages, the output is the first/last line, 08:04:20):

python driver.py
08:04:20
running build
running build_ext
building 'helloworld' extension
gcc ...
running install
running build
running build_ext
running install_lib
copying build/lib.linux-x86_64-3.7/helloworld.cpython-37m-x86_64-linux-gnu.so -> /home/wuebbel/.local/lib/python3.7/site-packages
running install_egg_info
Removing /home/wuebbel/.local/lib/python3.7/site-packages/HelloWorld-2.1-py3.7.egg-info
Writing /home/wuebbel/.local/lib/python3.7/site-packages/HelloWorld-2.1-py3.7.egg-info
08:04:20

Running the script again loads the correct module and shows the new time:

wuebbel@02e267406db3:~/Projekte/Mandelbrot Demos/helloworld$ python driver.py
08:16:58
...
08:16:58

It seems that my module never gets reloaded. What would be the correct way of doing this?

Frank Wuebbeling
  • 23
  • 6

1 Answers1

0

I have doubts about the feasability of reloading a module containing compiled code, reading this answer: https://stackoverflow.com/a/48814213/6451573 which suggests that PEP 489 changed module reloading (between python 3.6 and 3.7), possibly "breaking" shared library reload.

Due to limitations in shared library loading (both dlopen on POSIX and LoadModuleEx on Windows), it is not generally possible to load a modified library after it has changed on disk.

That could explain the problem you're having.

If this is an operational case, I suggest that you have a master python program which creates the package, then use separate python processes to load it.

subprog.py:

import helloworld
print(helloworld.hello('test'))

master.py

import os,subprocess,sys
def runit():
   subprocess.run([sys.executable,os.path.join(os.path.dirname(__file__),"subprog.py")])

runit()

os.system("touch helloworld.c")
os.system("python3 setup.py build")
os.system("python3 setup.py install --user")

runit()

Using sys.executable ensures that the same python executable is used for the master python process and the subprocess, so that's more transparent.

Jean-François Fabre
  • 137,073
  • 23
  • 153
  • 219
  • Thanks for your quick answer. I agree that DLL reload is hard, but it generally should work - after all, it's in the official repo, and nobody else seems to complain - so I still think I'm doing something wrong (and it works fine in Matlab for mex files, if I remember right). Unfortunately, your workaround does not help, result is the same as before. – Frank Wuebbeling Feb 09 '19 at 18:31
  • I have edited to propose a solution with a subprocess. Workaround too, slightly worse, but should work – Jean-François Fabre Feb 09 '19 at 20:17
  • Thanks for the update. Note that the API for subprocess.run has changed, I had to do something along the lines of ` print(subprocess.run([sys.executable,"subprocess.py"],capture_output=True)) `. While that works, it's not really what I need - Variables of the caller process get lost when calling a subprocess, I need interaction inside and more. I am currently concluding that I cannot get it to work the way I want it to. I am accepting your answer since that's the best one can probably do. – Frank Wuebbeling Feb 10 '19 at 06:07