3

I have got the following setup:

----------------------------
|     C++ Application      |
----------------------------
              |
          dlopen()s
              |
              v
----------------------------          -------------------
| C++ Plugin with embedded |--links-->| libpython3.8.so |
|    Python interpreter    |          -------------------
----------------------------
              |
            runs
              |
              v
----------------------------
|       Python script      |
----------------------------
              |
           imports
              |
              v
----------------------------
|   Python C extension     |
|   module (e.g., numpy)   |
----------------------------

Without any further measures, the import of the Python C extension module from within the Python script fails for the reasons detailed in this SO question, i.e. that the Python extension does not "see" the symbols in libpython.

When calling dlopen("libpython3.8.so", RTLD_LAZY | RTLD_GLOBAL) in the C++ plugin as suggested in answers to the aforementioned question, it works. Yet, intermingling build-time dynamic linking (i.e., specifing -lpython3.8) and "manual" dlopen() calls at run-time somehow does not seem right to me.

Therefore, I am looking for a more stringent solution. My first idea was to skip compile-time linking altogether and fully rely on dlopen() to load libpython. As an extra benefit, this would have allowed to make the Python installation user-selectable, by providing an option to specify the path to libpython in the config of my plugin. However, the approach does not work as even making use of RTLD_LAZY, variables in libpython have to be resolvable already at the moment when the C++ plugin is loaded by the application.

Vice versa, it could also be desirable to rely on build-time linking only. To my understanding, this would require a link-time option instructing to load libpython with the same symbol visibility as achieved by dlopen() with RTLD_GLOBAL. However, I do not know of any such option and was so far unable to find one.

My questions are:

  • Can you think of any better or more stringent solution than both linking libpython at build-time and dlopen()ing at runtime? It would be an extra bonus if an approach would allow to make libpython selectable via plugin config (independent of the LD_LIBRARY_PATH environment variable).
  • If you cannot think of any better solution either, what are the risks related to mixing build-time linking and dlopen()? In particular,
    • what would happen if - for any reason - another version of libpython was loaded with dlopen() than the one that was linked against?
    • are -lpython3.8 and dlopen("libpython3.8.so", ...) equivalent in the sense that they always resolve to the same library file?
Stehsegler
  • 41
  • 3
  • Could you provide compile/link commands that you use? `dlopen`ing libpython prior to loading the plugin should not be needed. – yugr Oct 11 '20 at 11:04
  • I'm calling `dlopen()` in the C++ plugin itself, not in host application. That was mistakable due to a now-fixed typo in the question - sorry for that. I'm compiling with `g++ -std=c++20 -c -g -I/usr/include/python3.8 -fPIC -MMD -MP -MF FILE.o.d -o FILE.o FILE.cpp` and linking with `g++ -std=c++20 -o plugin.so *.o -lcrypt -lpthread -ldl -lutil -lm -lpython3.8 -lvamp-sdk -shared -fPIC`. As mentioned in the question, the extra call to `dlopen()` is only necessary to be able to Python C extension modules. If the Python scripts do not include any of those, everything works just fine. – Stehsegler Oct 12 '20 at 16:21

0 Answers0