6

I am using the first step example in pybind11's documentation

#include <pybind11/pybind11.h>
int add(int i, int j)
{
    return i + j;
}
PYBIND11_MODULE(example, m)
{
    m.doc() = "pybind11 example plugin"; // optional module docstring
    m.def("add", &add, "A function which adds two numbers");
}

everything works fine, i can use it in python shell:

import  example
example.add(2, 3) #returns 5

Now i made a simple change to use float instead of int for input for add(), everything compiles. and i want to reload the module example so i can test the new float based add(). However, i can not figure out a way to reload the example module. importlib.reload does not work, %autorelaod 2 in IPython does not work either. both approached tested to work with pure python based modules, but not the c++ and pybind11 based modules.

Did I miss anything here? or it ought to be like this?

UPDATE: seems it is a known issue related to How to Reload a Python3 C extension module?

Python's import mechanism will never dlclose() a shared library. Once loaded, the library will stay until the process terminates.

pybind11 module and ctypes module seems to share the same traits here regarding how the module is loaded/imported. Also quote from https://github.com/pybind/pybind11/issues/2511:

The way C extensions are loaded by Python does not allow them to be reloaded (in contract to Python modules, where the Python code can just be reloaded and doesn't refer to a dynamically loaded library)

I now just wonder if there is a method to wrap this up in a more convenient way for reloading the module. E.g., spawn a subprocess for a new python shell that copies all C extensions related variable/module, and substitute the original one.

shelper
  • 10,053
  • 8
  • 41
  • 67
  • What do you mean reload the module? If you just replace the module generated by pybind11 and re-run your python script it will use the new module. – super Sep 19 '20 at 20:18
  • python has a reload() function, used to be a builtin and is nowadays in importlib. it can be used to reload previously imported modules, for which the code has changed in the meantime, again during the runtime of the python process. sometimes it works and sometimes not. usually what's needed, or stumbling it, is that objects that use some classes from the lib need to be recreated, just reloading the module does not change existing objects. – antont Sep 19 '20 at 21:31
  • 1
    @super i mean reload without exiting python shell, not running a script as a standalone – shelper Sep 20 '20 at 00:27
  • This was posted half a year ago. Have you found a solution to the problem? importlib.reload doesn't work for pybind modules. To answer: "Why not just re-run the entire Python script?" If there is 10 minutes start up time (e.g. loading 80gb from disk and preprocessing), I don't want to do that every time I change a comma in the C++ code. Being able to quickly reload the pybind module on the fly is crucial when developing and debugging. – James Apr 21 '21 at 16:36
  • @James No solution yet... – shelper Apr 27 '21 at 18:47

1 Answers1

0

Its seems no way straightforward. Since it's possible to manage standard shared library by manually dlopen() & dlclose(), you can change your PYBIND11_MODULE to a pre-defined function like

void __bind_module(void *bind_) {
  typedef void (*binder_t)(const char *, py::cpp_function);
  auto bind = (binder_t) bind_;
  bind("add", add);
}

and then write a manager module to attach/detach those libraries. Something like importlib from yourself.

Yuxai
  • 65
  • 1
  • 5
  • it would be greater if you can share more details, e.g., a minimal viable example. thx! – shelper Oct 03 '22 at 19:16
  • I think what hes trying to say is that you need 2 dlls. manager.dll < he will load/unload > worker.dll which is your code that you update/rewrite. You use your manager to load/unload it all in c++ without calls from python... maybe? – Dariusz Mar 03 '23 at 19:24