I have got the following setup:
----------------------------
| C++ Application |
----------------------------
|
dlopen()s
|
v
---------------------------- -------------------
| C++ Plugin with embedded |--links-->| libpython3.8.so |
| Python interpreter | -------------------
----------------------------
|
runs
|
v
----------------------------
| Python script |
----------------------------
|
imports
|
v
----------------------------
| Python C extension |
| module (e.g., numpy) |
----------------------------
Without any further measures, the import of the Python C extension module from within the Python script fails for the reasons detailed in this SO question, i.e. that the Python extension does not "see" the symbols in libpython.
When calling dlopen("libpython3.8.so", RTLD_LAZY | RTLD_GLOBAL)
in the C++ plugin as suggested in answers to the aforementioned question, it works. Yet, intermingling build-time dynamic linking (i.e., specifing -lpython3.8
) and "manual" dlopen()
calls at run-time somehow does not seem right to me.
Therefore, I am looking for a more stringent solution. My first idea was to skip compile-time linking altogether and fully rely on dlopen()
to load libpython. As an extra benefit, this would have allowed to make the Python installation user-selectable, by providing an option to specify the path to libpython
in the config of my plugin. However, the approach does not work as even making use of RTLD_LAZY
, variables in libpython
have to be resolvable already at the moment when the C++ plugin is loaded by the application.
Vice versa, it could also be desirable to rely on build-time linking only. To my understanding, this would require a link-time option instructing to load libpython
with the same symbol visibility as achieved by dlopen()
with RTLD_GLOBAL
. However, I do not know of any such option and was so far unable to find one.
My questions are:
- Can you think of any better or more stringent solution than both linking
libpython
at build-time anddlopen()
ing at runtime? It would be an extra bonus if an approach would allow to makelibpython
selectable via plugin config (independent of theLD_LIBRARY_PATH
environment variable). - If you cannot think of any better solution either, what are the risks related to mixing build-time linking and
dlopen()
? In particular,- what would happen if - for any reason - another version of libpython was loaded with
dlopen()
than the one that was linked against? - are
-lpython3.8
anddlopen("libpython3.8.so", ...)
equivalent in the sense that they always resolve to the same library file?
- what would happen if - for any reason - another version of libpython was loaded with