Python not able to find the external dependent dll

Question

I am using Python3.6. I have created a C++ extension using (pybind11)[https://github.com/pybind/pybind11]. I copied the compiled *.pyd file along with the dependent dll to the site packages. But when I try to load any functions from the external DLL, python complains that the function is not present. If I want to access the function, I need write

sys.path.append(r'C:\Users\test\AppData\Local\Programs\Python\Python36\Lib\site-packages\CppProject')

or I need to add the same path to the PYTHONPATH environment variable.

Why Python is not able to load the function even though it is present in the same path as the pyd? I don't want to append the sys path everytime I need to use the module or use the environment variable? Is there any way to avoid this? Is there any way to add this path to the sys automatically whenever the user import the module?

Example:

CppExport.dll

#ifdef CPPEXPORT_EXPORTS
#define CPPEXPORT_API __declspec(dllexport)
#else
#define CPPEXPORT_API __declspec(dllimport)
#endif

extern "C" CPPEXPORT_API double sin_impl(double x);


const double e = 2.7182818284590452353602874713527;
double sin_impl(double x){
    return (1 - pow(e, (-2 * x))) / (2 * pow(e, -x));
}

CppProject.pyd

PYBIND11_MODULE(CppProject, m) {

    m.def("sin_impl", &sin_impl, R"pbdoc(
        Compute a hyperbolic tangent of a single argument expressed in radians.
    )pbdoc");

#ifdef VERSION_INFO
    m.attr("__version__") = VERSION_INFO;
#else
    m.attr("__version__") = "dev";
#endif
}

Setup.py

from setuptools import setup
import distutils
import sys

from setuptools.dist import Distribution

from distutils.sysconfig import get_python_lib
relative_site_packages = get_python_lib().split(sys.prefix+os.sep)[1]
date_files_relative_path = os.path.join(relative_site_packages, "CppProject")

class BinaryDistribution(Distribution):
    """Distribution which always forces a binary package with platform name"""
    def has_ext_modules(foo):
        return True


setup(
    name='CppProject',
    version='1.0',
    description='CppProject Library',
    packages=['CppProject'],
    package_data={
        'CppProject': ['CppProject.pyd'],
    },
    data_files = [(date_files_relative_path, ["CppExport.dll"])],
    distclass=BinaryDistribution
)

In Python:

from CppProject import sin_impl

Error:

ImportError: cannot import name 'sin_impl'

Full Code is present in Github

Please try to provide a [mcve], or at least as much concrete, relevant information as possible. There is little point in using a bounty to draw attention to a question that barely contains information on the issue. How do you try to import the extension? What error do you get? What is the content of ``sys.path`` to begin with. — MisterMiyagi, Jul 13 '21 at 13:11
How are you loading functions from the dll? Can you post the piece of code that performs this task? — Max Shouman, Jul 13 '21 at 13:24
@MisterMiyagi added the minimal example and the full code is present in github — samnaction, Jul 13 '21 at 17:19
@MaxShouman the dll function is loaded using the PyBind11 Module — samnaction, Jul 13 '21 at 17:20

Teddy · Answer 1 · 2021-07-13T13:49:53.237

-1

Sorry for the previous reply here is some better advises :

You want to distribute your library, to do so you need to create setup.py and init.py. Once this done you will be able to install your package using python setup.py install.

For me the setup.py look like :

README_rst = ''
from distutils.core import setup
with open('README.rst', mode='r', encoding='utf-8') as fd:
    README_rst = fd.read()

setup(
    name='MyStack',
    version='0.0.1',
    description='Cool short description',
    author='Author',
    author_email='author@mail.com',
    url='repo.com',
    packages=['Pkg'],
    long_description=README_rst,
    include_package_data=True,
    classifiers=[
        # Trove classifiers
        # The full list is here: https://pypi.python.org/pypi?%3Aaction=list_classifiers
        'Development Status :: 3 - Alpha',
    ]
)

In the init.py you will have to find your library and import it. Here is an example how Qt does :

def find_qt():
    import os

    path = os.environ['PATH']

    dll_dir = os.path.dirname(__file__) + '\\Qt\\bin'
    if os.path.isfile(dll_dir + '\\Qt5Core.dll'):
        path = dll_dir + ';' + path
        os.environ['PATH'] = path
    else:
        for dll_dir in path.split(';'):
            if os.path.isfile(dll_dir + '\\Qt5Core.dll'):
                break
        else:
            raise ImportError("unable to find Qt5Core.dll on PATH")

    try:
        os.add_dll_directory(dll_dir)
    except AttributeError:
        pass


find_qt()
del find_qt

Hope this help

edited Jul 13 '21 at 13:49

answered Jul 13 '21 at 13:06

Teddy

57
1
7

Can you explain why the pyd file, not able to find the dll even if it is present in the same path? – samnaction Jul 14 '21 at 03:10
I don't fully understand the way programs load dll but it doesn't necessarily start on the programs path. It will start looking in the directory added in the environment variable. Also a .pyd file is just a dll with specific function like PyInit (those are declared by pybind11 in your case) and a specific name, it contains dependencies that your os is supposed to satisfy. In your edited post it seems that it loads the dll, but it cannot get the function sin_impl. Does’ this code works by adding the path to the environment variable ? – Teddy Jul 15 '21 at 06:41
Your project work on my side, I had to change some path you should consider use ```$(SolutionDir)```, ```$(Platform)``` variable this will limit the number of change if you copy/paste your project somewhere else. To make it work I only add ```python36.dll``` in the path of your .pyd file. This is because I didn't have python36 in my PATH environment variable. Also if you only need to build a python module you don't have to pass through a dll you can directly code your function in the CppProject – Teddy Jul 15 '21 at 07:13
Adding the dll path to the environment work as I mentioned in the question. Even though the dll is present in site-package. – samnaction Jul 15 '21 at 10:52
Were you able to call the dll function sin_impl ? I can import CppProject but cant call the sin_impl function. – samnaction Jul 15 '21 at 10:53
Yes I was able to call the function. As I told you you cannot place your dll in site-packages and hope python will find it. If you don't want to have the dll and .pyd in your working directory you have to "build" a distribution. To load your dll I create a folder with : ```CppExport.dll```, ```CppProject.lib```, ```CppProject.pyd```, ```python36.dll```. I open new console ```cd``` to this folder call ```python``` and execute the following commands ```import CppProject``` then ```CppProject.sin_impl(5)``` which returns ```74.20321``` – Teddy Jul 15 '21 at 12:19
I was not copying, I also installed using setup.py. The python was able to find CppProject, but not the function from CppExport which is present in the same path as CppProject. If I understand, the dll path has to explicitly added to the Python even though they are in the site-package along with the pyd. Is that right? – samnaction Jul 15 '21 at 12:39
If you install your package using ```python setup.py install``` it should work without adding the folder to the Python even. Normally the distutils should do it for you. In your example you install the package ```CppProject``` containing the module ```CppProject``` which contains sin_impl. Could you try to call the function without installing using setup ? – Teddy Jul 15 '21 at 12:59

Max Shouman · Answer 2 · 2021-07-15T21:26:27.217

The fact that your code works when you explicitly add the directory to sys.path is the key to understand what's happening.

Since site-packages is one of the locations searched by the interpreter when importing modules, this statement:

from CppProject import sin_impl

is actually searching for a module named sin_impl inside the CppProject folder.

Instead you should do:

from CppProject.CppProject import sin_impl

which points to the actual module of the same name.

This actually doesn't require the presence of __init__.py inside CppProject to qualify it as a Python package, since Python 3.3+ implements implicit namespace packages.

However, when you are building a complex program with many dependencies the package constructor enables you to add some kind of initialization to be performed before any regular module is executed.

Would he put the pyd file next to the python script and execute the python script would execute fine. Your answer doesn't make sense. It not at all a semantic issue. — Olórin, Jul 30 '23 at 20:45

Python not able to find the external dependent dll

2 Answers2