Python3 shared extension doesn't link against library dependency

Question

I'm creating a shared Python extension for my library and I'm using distutils to build it.

These are the relevant sections of my setup.py:

import distuitls.core as dc
from os.path import join as path_join

module = dc.Extension(module_name, 
                      sources = [path_join(meson_src_root, "py3_bindings", "module.c")], 
                      include_dirs = [path_join(meson_src_root, "include")],
                      libraries = ["bbmputil"],
                      runtime_library_dirs = [meson_build_root])

dc.setup(name = module_name,
         version = module_version,
         description = "Python3 bindings for the bbmp_utils library",
         ext_modules = [module])

---

Running $ setup.py build results in the shared extension module being built successfully, but it isn't getting linked against the "bbmputil" library.

$ ldd build/lib.linux-x86_64-3.8/bbmp_utils.cpython-38-x86_64-linux-gnu.so

linux-vdso.so.1 (0x00007ffc85ce1000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f49f0d70000)
/usr/lib64/ld-linux-x86-64.so.2 (0x00007f49f0f74000)

libbbmputil.so is nowhere to be found, despite being specified in the libraries kwarg of Extension().

It does exist in the location specified in the runtime_library_dirs kwarg.

---

This leads to the python interpreter raising an ImportError exception when a symbol from the non-linked library is referenced in the extension:

$ env PYTHONPATH="sharedextension_build_path" python3

>>> import bbmp_utils
ImportError: /home/bogdan/dev/bbmp_utils/build_dbg/build/lib.linux-x86_64-3.8/bbmp_utils.cpython-38-x86_64-linux-gnu.so: undefined symbol: bbmp_vertflip

where bbmp_vertflip is a symbol defined in the library that doesn't seem to be linked for some reason.

---

The two C compiler invocations look as follows:

gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fPIC -I/home/bogdan/dev/bbmp_utils/include -I/usr/include/python3.8 -c /home/bogdan/dev/bbmp_utils/py3_bindings/module.c -o build/temp.linux-x86_64-3.8/home/bogdan/dev/bbmp_utils/py3_bindings/module.o

gcc -pthread -shared -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now build/temp.linux-x86_64-3.8/home/bogdan/dev/bbmp_utils/py3_bindings/module.o -L/usr/lib -Wl,--enable-new-dtags,-R/home/bogdan/dev/bbmp_utils/build_dbg -lbbmputil -o build/lib.linux-x86_64-3.8/bbmp_utils.cpython-38-x86_64-linux-gnu.so

In the 2nd invocation both -lbbmputil as well as -R are passed properly when building the shared extension so I'm out of ideas.

---

Minimal example producing the same behavior

Attempting to build a module that utilizes functions and other symbols from the math shared library:

#!/usr/bin/env python3

import distutils.core as dc

module = dc.Extension('example',
                      sources = ['example.c'],
                      libraries = ['m'])
dc.setup(name = 'example',
         version = '0.1',
         ext_modules = [module])

$ ./setup.py build

$ ldd .../.../example.cpython-38-x86_64-linux-gnu.so

linux-vdso.so.1 (0x00007ffd0b9e5000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007fab528e8000)
/usr/lib64/ld-linux-x86-64.so.2 (0x00007fab52aec000)

---

Again, libm.so dependency is nowhere to be found.

---

Environment:

• python3 3.8.1
• linux 5.4.6
• gcc 9.2.0
• ld 2.33.1
• ldd 2.3.0

Answer 1

UPDATE : The problem in this case is the linker optimization option --as-needed that is enabled by default, see Missing a library in ldd after using gcc -l

Adding --no-as-needed fixes this error

For debugging linker errors you can use LD_DEBUG=files,libs /usr/local/ABC/bin/ABC where ABC is the executable that throws linker errors at runtime, cf http://www.bnikolic.co.uk/blog/linux-ld-debug.html and libm.so.6: cannot open shared object file: No such file or directory On linux you locate a .so with i.e. locate libm (i think you know this)

As the linking is dynamically it is an option to specify the path where your .so files can be found using the library_dirs option of disutils.core that is the -L or equivalently LD_LIBRARY_PATH gcc linker option and for reasons of debugging and testing i would use the absolute path (https://docs.python.org/2/distutils/apiref.html)

In your python minimal example the code is then :

#!/usr/bin/env python3

import distutils.core as dc

module = dc.Extension('example',
                      sources = ['example.c'],
                      library_dirs = ['/usr/lib/x86_64-linux-gnu/libm.so'],
                      libraries = ['m'])
dc.setup(name = 'example',
         version = '0.1',
         ext_modules = [module])

You use the -R linker flag to specify the rpath in your gcc invokation, cf Shared library dependencies with distutils and What does the gcc -R parameter do? . In https://www.mpcdf.mpg.de/services/computing/software/libraries/static-and-dynamic-linking-on-linux-systems is a description of the linking process. It is said that LD_LIBRARY_PATH or equivalently the -L gcc linker option overrides the rpath and that it should be avoided, however you should give it a try anyway ...

Another possiblity for this behavior could be permission problems, i.e. when you execute example does it have the permission to access libm cf https://unix.stackexchange.com/questions/303292/permission-denied-on-some-shared-libraries