0

I am porting a swig extension module that was designed for Python 2 to Python 3 under Windows 10 with a stock Python 3.9.1. The module builds properly and depends on several Windows dynamic load libraries (*.dll) that I have packaged into the data directory.

Importing the module fails, and using gflags to track down the problem shows that the dependent dll's are not being loaded as the search path does not include the module name: lib/site-packages/dbxml. When I manually move the dll's into the lib/site-packages, they can be located and successfully loaded.

I am confused on how to modify setup to generate a DLL path that includes the module name. I saw this post that seemed to imply that I could simply use the data_files option, but that does not appear to work in my case.

Here is what I am doing with what I believe to be the relevant variables replaced with their values:

  setup(name = "dbxml",
    version = "6.1.4",
    description = "Berkeley DB XML Python API",
    long_description = """removed...""",
    author = "Oracle",
    author_email = "berkeleydb-info_us@oracle.com",
    url = "http://www.oracle.com",
    py_modules = ["dbxml"],
    ext_modules = [Extension("_dbxml", ["dbxml_python3_wrap.cpp"],
                             include_dirs = INCLUDES,
                             library_dirs = ['../../../lib', '../../build_windows/Release', 
                                             '../../../db-6.2.23/build_windows/Release', 
                                             '../../../xqilla/lib', 
                                             '../../../xerces-c-src/Build/Win32/VC10'],
                             define_macros = DEFINES,
                             libraries = ['libdbxml61', 'libdb62', 'xqilla23', 'xerces-c_3'],
                             extra_compile_args = ['/GR', '/EHsc']
                             )],
    # The DLLs below are copied into lib/site-packages/dbxml
    # but the DLL search path is:
    # C:\Users\[...omitted...]\python3.9\lib\site-packages;
    # C:\Users\[...omitted...]\python3.9;
    # C:\WINDOWS\SYSTEM32
    # and they are not found.
    data_files = [('lib/site-packages/dbxml',
                   ['../../../bin/libdbxml61.dll',
                    '../../../bin/libdb62.dll', 
                    '../../../bin/xqilla23.dll', 
                    '../../../bin/xerces-c_3_1.dll', 
                    '../../build_windows/zlib1.dll', 
                    '../../build_windows/zlibwapi.dll'
                   ])
                 ]
   )

My understanding is that Extension's runtime_library_dirs keyword argument only controls the runtime library path on UNIX-like operating systems and is ignored on Windows 10. Obviously, I am doing something wrong. Any suggestions on how to fix this would be appreciated.

Thank you - Marie

Marie R
  • 270
  • 3
  • 9
  • After installing the module, what is its layout? Meaning what's the relative path relation between *\_dbxml.pyd* and the *.dll*s that it depends on? Is *dbxml.py* autogenerated? Also, those path separators look awful. – CristiFati Mar 13 '21 at 05:12
  • @CristiFati, the layout is: dbxml.py dbxml-6.1.4-py3.9.egg-info in the site-packages directory with all of the dlls in site-packages/dbxml: libdb62.dll, xerces-c_3_1.dll, zlib1.dll, libdbxml61.dll, xqilla23.dll, zlibwapi.dll. The dbxml.py was generated by swig. Sorry for the ugly path names,they were generated by code above the setup and I should have cleaned them up a bit. – Marie R Mar 13 '21 at 15:23

1 Answers1

2

It appears that the DLL library loading behavior changed in Python 3.8. The path for DLLs in Windows must be set explicitly using os module function add_dll_directory. I was able to resolve this by adding:

# add peer directory to dll search path
import os
dll_dir = os.path.join(os.path.dirname(__file__), "dbxml")
os.add_dll_directory(dll_dir)

There may be a better way to do this, but this will let the extension module load dyanmicaly linked libraries from a peer directory in site-packages.

Marie R
  • 270
  • 3
  • 9