How do I use the correct dll files to enable 3rd party C libraries in a Cython C extension?

Question

I have a C function that involves decompressing data using zstd. I am attempting to call that function using Cython.

Using this page from the docs as a guide I can compile and run the code below with no problem.

(I don't actually use the zstd lib here)

// hello.c
#include <stdio.h>
#include <zstd.h>

int hello() {
   printf("Hello, World!\n");
   void *next_in = malloc(0);
   void *next_out = malloc(0);
   return 0;
}

# Hello.pyx

cdef extern from "hello.c":
  int hello()

cpdef int callHello():
  hello()

# hello_wrapper.setup.py

from setuptools import setup, Extension
from Cython.Build import cythonize

ext_modules = [
    Extension(
        "hello_wrapper",
        ["hello_wrapper.pyx"],
        libraries=["zstd"],
        library_dirs=["path/to/zstd/lib"],
        include_dirs=['path/to/zstd/include'],
    )
]

setup(
    ext_modules = cythonize(ext_modules, gdb_debug=True)
)

Using the commands as follows I get the expected output:

>py hello_wrapper.setup.py build_ext --inplace
>py
Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:20:19) [MSC v.1925 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import hello_wrapper
>>> hello_wrapper.callHello()
Hello, World!
0

However when I modify hello.c to actually use the zstd library:

// hello.c
#include <stdio.h>
#include <zstd.h>

int hello() {
   printf("Hello, World!\n");
   void *next_in = malloc(0);
   void *next_out = malloc(0);
   size_t const dSize = ZSTD_decompress(next_out, 0, next_in, 0); //the added line
   return 0;
}

While hello_wrapper.setup.py compiles fine, when I get to the import statement, I get the following error:

>>> import hello_wrapper
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing hello_wrapper: The specified module could not be found.

From reading This SO article, I gather that this error means I'm not correctly pointing to or perhaps creating in the first place the required DLL files for zstd.lib to work its magic. Is this correct? If so, how might I do that? If not, what is the problem?

Answer 1

We link our cython-extension against a windows-dll, that means:

• *.lib-file (i.e. zstd.lib) is needed in "path/to/zstd/lib" during the compile time
• *.dll-file (i.e. zstd.dll) is needed somewhere where Windows can find it when the module is imported.

Normally, Windows will not look in the "path/to/zstd/lib". And so we get a somewhat cryptic error message:

ImportError: DLL load failed: The specified module could not be found.

Which doesn't mean there is something wrong with the module - it just happens to depend on a dll which cannot be found.

While linux has -rpath-option for the linker with which "path/to/zstd/lib" could be passed (it can be added with runtime_library_dirs-argument to Extension), there is no such option on Windows.

The dll-search-algorithmus for Windows can be found here. In a nutshell, dll is searched in (possible in another order as presented here)

• The directory from which the module is loaded.
• The current working directory.
• system-directory (e.g. C:\Windows\System32)
• windows-directory(e.g. C:\Windows)
• directories that are listed in the PATH-variable
• others

However, since Python3.8 the above default algorithm isn't used for CPython: The current working directory and the PATH-variable are no longer used during the call, but there is os.add_dll_directory which could be used to add paths which will be used when resolving dependencies.

Putting the dll into system- or windows-directory doesn't sound too appealing, which leave us with the following options:

• (the easiest?) to copy the zstd.dll next to the compiled extension
• to use os.add_dll_directory to add location to the search for Python>=3.8
• to add the zstd-path to the PATH-variable, e.g. set PATH="path/to/zstd/lib";%PATH% (for Python<3.8)

---

Another option is somewhat more tricky: Given that

If a DLL with the same module name is already loaded in memory, the system checks only for redirection and a manifest before resolving to the loaded DLL, no matter which directory it is in. The system does not search for the DLL.

we can use ctypes to "preload" the right dll, which will be used (without the need to search for it on the disc) when the wrapper-module is imported, i.e.:

import ctypes; 
ctypes.CDLL("path/to/zstd/lib/zstd.dll"); # we preload with the full path

import hello_wrapper  # works now!

---

The above applies if the extension is built and used on the same system (e.g. via build_ext --inplace). installation/distribution is somewhat more cumbersome (this is covered by this SO-post), one idea would be:

• to put *.h-, *.lib- and *.dll-files into 'package_data' (it seems to happen automatically anyway)
• the right relative library_path (or programmatically the absolute path) can be set in the setup.py so *.lib is found by the linker.
• dll will be put next to the compiled *.pyd-file in the installation.

An example could be the following more or less minimal setup.py, where everything (pyx-file, h-files, lib-file, dll-file) are put into a package/folder src/zstd:

from setuptools import setup, Extension, find_packages
from Cython.Build import cythonize

ext_modules = [
    Extension(
        "zstd.zstdwrapper",
        ["src/zstd/zstdwrapper.pyx"],
        libraries=["zstd"],
        library_dirs=["src/zstd"],
        include_dirs=[], # set automatically to src/zstd during the build
    )
]

print(find_packages(where='src'))

setup(
    name = 'zstdwrapper',
    ext_modules = cythonize(ext_modules),
    packages = find_packages(where='src'),
    package_dir = {"": "src"},
)

And now it can be installed with python setup.py install or used to create e.g. a source-distribution via python setup.py sdist which then can be installed via pip.