I am working on a C++ extension for Python using PyBind11. My extension depends on CGAL, which depends on MPFR and GMP. CGAL is header-only, so there are no problems there, but MPFR and GMP compile to shared libraries (DLL files on Windows).
Everything compiles and links fine on every operating system. On Windows, I have to supply the CMAKE_PREFIX_PATH option to CMake to help it find the location of the libraries (which in my case are installed via Miniconda, so they live at C:\Miniconda3\Library), so it correctly locates CGAL, GMP and MPFR. After compilation and linking, everything runs fine on Linux and Mac, but on Windows, there is a problem. Simply, when I try to import my extension module in Python, I am met with the error
ImportError: DLL load failed while importing <mylibrary>: The specified module could not be found.
I understand that this error means that a DLL that my extension depends on (in this case, MPFR and GMP) can not be located. If I add the directory containing these DLLs to my PATH, I would expect this to fix the problem, but apparently, Python no longer searches the PATH for DLLs, according to a few posts I've read (example 1, example 2, example 3, python docs). Instead, we are expected to call os.add_dll_directory() to tell Python where to find the DLLs. This makes absolutely no sense to me, because as the library writer, I am not aware of, and I don't believe I should be dependent on, the location in which a user decides to install a library.
For example, maybe the user installs MPFR manually, or maybe they install it via Conda, or with VCPKG, who knows. Each of these will place it in a very different location, unknown to me. How am I supposed to know, in my Python module, where to find these libraries?
This question is very similar, but the DLLs are packaged with the libraries source code, so their location is obvious, and os.add_dll_directory() is a sensible solution. In my case, I am depending on external DLLs that the user could have installed anywhere.
I will enumerate the suboptimal solutions that I have found so far. None of them are particularly good.
Solution 1 To confirm my suspicion that these are the libraries that can not be found, I was able to fix the problem by manually copying the relevant DLL files into the same directory as my compiled extension module. Of course, this solution is manual and not robust, and I would have to do it each time I make a new build. It is also not a nice solution to have to describe to users who would like to compile my library.
Solution 2 When building wheels of my library, I found the Python script delvewheel, which searches for the DLLs that a wheel depends on and adds them to the wheel (it actually finds the DLLs because it looks at PATH!!!) This appears to solve the problem in the case of distributing a wheel, which is nice, but it does not help in the case where a user wants to manually compile and test my library locally, unless they too go through the steps to build the wheel, repair it with delvewheel, and then install it, which does not make sense for a local development/testing workflow.
Solution 3 I can hardcode the location of the DLL files into a call to os.add_dll_directory(). This works, but of course, this is terrible because it depends on where I have installed them! Clearly not suitable for distributing the code to others.
Solution 4 I could require the user to specify a special environment variable that points to the location of the DLLs, and then pass the value of this environment variable to os.add_dll_directory(). This means that the user has to specify a special environment variable every time they want to run my library. Not great. Also, it would not work if several of the dependencies were in different locations.
Solution 5 I could also just parse the PATH and pass every single location in it to os.add_dll_directory(), like so.
if "add_dll_directory" in dir(os):
for d in os.environ['path'].split(';'):
if os.path.isdir(d):
os.add_dll_directory(d)
This works, but it would seem to circumvent the entire purpose of Python not looking in the PATH and using os.add_dll_directory() in the first place. Is there a problem with this? The python changelog that introduced this seems to suggest that security might be an issue?
