_ctypes.cpython-39-x86_64-linux-gnu.so: undefined symbol: PyFloat_Type in embedded Python loaded with dlopen

Question

I am using embedded Python (3.9) in ubuntu 20.04 and trying to import ctypes which produces the error _ctypes.cpython-39-x86_64-linux-gnu.so: undefined symbol: PyFloat_Type.

I am compiling a shared object, which is loaded dynamically using dlopen().

CMake is used to build the shared object. I am stating Python3 dependency like so: find_package(Python3 REQUIRED COMPONENTS Development Development.Embed) and link using target_link_libraries(${target_name} Boost::filesystem Python3::Python)

If I understand correctly, this tells CMake to link directly with libpython3.9.so (I also tried to explicitly state linking to libpython3.9.so, but that did not solve the issue). I do see that libpython3.9.so exports PyFloat_Type and that _ctypes.cpython-39-x86_64-linux-gnu.so does not.

The import is simply done by the PyRun_SimpleString() function: PyRun_SimpleString("import ctypes").

I should state that I have seen on the web some solutions, but none of them worked (like exporting LD_FLAGS="-rdynamic", but does also did not help).

I should also point out that importing using the interpreter (python3.9) works well.

Here is the build command generated by CMake: /usr/bin/c++ -fPIC -g -Xlinker -export-dynamic -shared -Wl,-soname,mytest.python3.so -o mytest.python3.so CMakeFiles/mytest.python3.dir/[mydir]/[myobjects].o /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.71.0 /usr/lib/x86_64-linux-gnu/libpython3.9.so /usr/lib/x86_64-linux-gnu/libpython3.9.so

Thanks for any help in advance!

Answer 1

When a C-extension is imported in CPython on Linux, dlopen is used under the hood (and per default with RTLD_LOCAL-flag).

A C-extension usually needs functionality from the Python-library (libpythonX.Y.so), like for example PyFloat_Type. However, on Linux the C-extension isn't linked against the libpythonX.Y.so (the situation is different on Windows, see this or this for more details) - the missing function-definition/functionality will be provided by the python-executable.

In order to be able to do so, the executable must be linked with -Xlinker -export-dynamic, otherwise loader will not be able to use the symbols from the executable for shared objects loaded with dlopen.

Now, if the embedded python is not the executable, but a shared object, which loaded with dlopen itself, we need to ensure that its symbols are added to the dynamic table. Building this shared object with -Xlinker -export-dynamic doesn't make much sense (it is not an executable after all) but doesn't break anything - the important part, how dlopen is used.

In order to make symbols from text.python.so visible for shared objects loaded later with dlopen, it should be opened with flag RTLD_GLOBAL:

RTLD_GLOBAL The symbols defined by this shared object will be made available for symbol resolution of subsequently loaded shared objects.

i.e.

shared_lib = dlopen(path_to_so_file, RTLD_GLOBAL | RTLD_NOW);

Warning: RTLD_LAZY should not be used.

The issue with RTLD_LAZY is that C-extensions do not have dependency on the libpython (as can be seen with help of ldd), so once they are loaded and a symbol (e.g. PyFloat_Type) from libpython which is not yet resolved must be looked up, dynamic linker doesn't know that it has to look into the libpython.

On the other hand with RTLD_NOW, all symbols are resolved and are visible when a C-extension is loaded (it is the same situation as in the "usual" case when libpython is linked in during the linkage step with -Xlinker -export-dynamic) and thus there is no issue finding e.g. PyFloat_Type-symbol.

---

As long as the embedded python is loaded with dlopen, the main executable doesn't need to be built/linked with -Xlinker -export-dynamic.

However, if the main executable is linked against the embedded-python-shared-object, -Xlinker -export-dynamic is necessary, otherwise the python-symbols won't be visible when dlopen is used during the import of c-extension.

---

One might ask, why aren't C-extension linked against libpython in the first place?

Due to used RTLD_LOCAL, every C-extension would have its own (uninitialized) version of Python-interpreter (as the symbols from the libpython would not be interposed) and crash as soon as used.

To make it work, dlopen should be open with RTLD_GLOBAL-flag - but this is not a sane default option.