A project I contribute to is using a Rust language bindings library called UniFFI to create Python language bindings from Rust code. The crate produces a native library (say libtinymath.dylib) and a Python module (say tinymath.py which imports and uses the native library.
The python package layout looks like this:
❯ tree
.
├── src
│ ├── tinymath
│ │ ├── __init__.py
│ │ ├── tinymath.py
│ │ └── libtinymath.dylib
├── tests
│ └── test_tinymath.py
├── setup.py
└── tox.ini
My setup.py build file ensures that the library gets copied in the package when buildtools creates it:
# setup.py
from setuptools import setup, find_packages
setup(
name='tinymath',
version='0.0.1',
packages=find_packages(where="src"),
package_dir={"": "src"},
package_data={"tinymath": ["*.dylib"]},
include_package_data=True,
zip_safe=False,
)
The problem is that after installing the library using pip, the import statements fail stating that Python cannot find the native library in the default path. This makes sense because the library is not indeed in the path Python uses (I can see from the error returned). I learned through searching for an answer for this issue that in fact Python search path is different for normal Python modules (which use PYTHONPATH) than they are for native libraries. This is what is giving me trouble. I can change the location where Python looks for modules if I want to, but I cannot seem to adjust where Python looks for Native libraries.
Core Problem
So I set out to change the path the package uses to search for native libraries on Linux and macOS; all I need is to ensure it looks "inside itself" for it (the native lib is indeed sitting right next to the module). I'm not sure if this should be specified in the setup.py file or through some other method. This is the core question I'm asking here: How can I tell the package to look in the current directory when it goes looking for native libs?
Alternative Approaches
Another avenue one can use to make this work is to declare exactly where to find the library from within the module (tinymath.py in the example above). But I'd rather not use this approach, mostly because this implies going in and playing with the auto-generated language bindings module provided by UniFFI (and which they recommend to not touch). It does work however, validating the idea that the problem is only with the search path and that everything else works well.
Using this approach, my current fix is to change the library loading function from its auto-generated version:
def loadIndirect():
if sys.platform == "linux":
libname = "lib{}.so"
elif sys.platform == "darwin":
libname = "lib{}.dylib"
elif sys.platform.startswith("win"):
# As of python3.8, ctypes does not seem to search $PATH when loading DLLs.
# We could use `os.add_dll_directory` to configure the search path, but
# it doesn't feel right to mess with application-wide settings. Let's
# assume that the `.dll` is next to the `.py` file and load by full path.
libname = os.path.join(
os.path.dirname(__file__),
"{}.dll",
)
return getattr(ctypes.cdll, libname.format("tinymath"))
To the modified version:
def loadIndirect():
if sys.platform == "linux":
# libname = "lib{}.so"
libname = os.path.join(os.path.dirname(__file__), "lib{}.so")
elif sys.platform == "darwin":
# libname = "lib{}.dylib"
libname = os.path.join(os.path.dirname(__file__), "lib{}.dylib")
elif sys.platform.startswith("win"):
# As of python3.8, ctypes does not seem to search $PATH when loading DLLs.
# We could use `os.add_dll_directory` to configure the search path, but
# it doesn't feel right to mess with application-wide settings. Let's
# assume that the `.dll` is next to the `.py` file and load by full path.
libname = os.path.join(
os.path.dirname(__file__),
"{}.dll",
)
return getattr(ctypes.cdll, libname.format("tinymath"))
