5

As far as I understand, one of the main advantages of distributing Python packages through wheels is that I can include extension modules in a compiled form. Then, the user of the package is not required to have a system that allows compilation of the source code.

Now I managed to build a wheel for my package that includes a Fortran extension module. The computer on which I built has Windows7 64, and Python 3.6.

In order to get everything running, I followed this very helpful guideline (many thanks to Michael Hirsch). One of the steps was to install MinGW-64 with the following settings: Architecture: x86_64, Threads: posix, Exception: seh.

I then installed the Python package on another testing machine (Win10 64, Python 3.6) from that wheel:

D:\dist2>pip install SMUTHI-0.2.0a0-cp36-cp36m-win_amd64.whl
Processing d:\dist2\smuthi-0.2.0a0-cp36-cp36m-win_amd64.whl
Requirement already satisfied: scipy in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: sympy in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: argparse in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: numpy in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: matplotlib in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: pyyaml in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: six>=1.10 in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Requirement already satisfied: python-dateutil in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Requirement already satisfied: pytz in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Requirement already satisfied: cycler>=0.10 in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Installing collected packages: SMUTHI
Successfully installed SMUTHI-0.2.0a0

However, when I started a test run of the program, I encountered the following error:

D:\dist2>smuthi example_input.dat
Traceback (most recent call last):
  File "c:\programdata\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\programdata\anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\Scripts\smuthi.exe\__main__.py", line 5, in <module>
  File "c:\programdata\anaconda3\lib\site-packages\smuthi\__main__.py", line 4, in <module>
    import smuthi.read_input
  File "c:\programdata\anaconda3\lib\site-packages\smuthi\read_input.py", line 3, in <module>
    import smuthi.simulation
  File "c:\programdata\anaconda3\lib\site-packages\smuthi\simulation.py", line 8, in <module>
    import smuthi.t_matrix as tmt
  File "c:\programdata\anaconda3\lib\site-packages\smuthi\t_matrix.py", line 6, in <module>
    import smuthi.nfmds.t_matrix_axsym as nftaxs
  File "c:\programdata\anaconda3\lib\site-packages\smuthi\nfmds\t_matrix_axsym.py", line 11, in <module>
    import smuthi.nfmds.taxsym
ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden.

The extension .pyd file (taxsym.cp36-win_amd64.pyd) was at its place - just Python couldn't load it.

Next, I uninstalled MinGW from the testing machine and reinstalled MinGW-64 with the same settings that I had used on the building machine (see above). Afterwards, I could run the program, and Python was able to correctly load the extension module.

My question is: Does anybody have an idea why the error occurred in the first place? And how can I avoid that the user of my Python package has to have a specific version of MinGW installed (or even any) for the package to work properly?

Edit: A small example that reproduces the error:

Minimal example

File structure:

setup.py
example/
    __init__.py
    run_hello.py
    extension_package/
        __init__.py             
        fortran_hello.f90

The setup.py reads:

import setuptools
from numpy.distutils.core import Extension
from numpy.distutils.core import setup

setup(
   name="example",
   version="0.1",
   author="My Name",
   author_email="my@email.com",
   description="Example package to demonstrate wheel issue",
   packages=['example', 'example.extension_package'],
   ext_modules=[Extension('example.extension_package.fortran_hello',
                          ['example/extension_package/fortran_hello.f90'])],
)

The run_hello.py reads:

import example.extension_package.fortran_hello
example.extension_package.fortran_hello.hello()

The fortran_hello.f90 reads:

subroutine hello
print *,"Hello World!"
end subroutine hello

Creation of the wheel

I ran python setup.py bdist_wheel which resulted in the file example-0.1-cp36-cp36m-win_amd64.whl

Installation of the package on machine with correct MinGW version

D:\dist>pip install example-0.1-cp36-cp36m-win_amd64.whl
Processing d:\dist\example-0.1-cp36-cp36m-win_amd64.whl
Installing collected packages: example
Successfully installed example-0.1

D:\dist>python
Python 3.6.0 |Anaconda 4.3.1 (64-bit)| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import example.run_hello
 Hello World!
>>> exit()

This is as it should be.

Installation of the package on machine without correct MinGW version

To reproduce the error, I renamed the MinGW folder on the testing machine to some other name and then:

D:\dist>pip install example-0.1-cp36-cp36m-win_amd64.whl
Processing d:\dist\example-0.1-cp36-cp36m-win_amd64.whl
Installing collected packages: example
Successfully installed example-0.1

D:\dist>python
Python 3.6.0 |Anaconda 4.3.1 (64-bit)| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import example.run_hello
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ProgramData\Anaconda3\lib\site-packages\example\run_hello.py", line 1, in <module>
    import example.extension_package.fortran_hello
ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden.
Community
  • 1
  • 1
Amos Egel
  • 937
  • 10
  • 24
  • Which way are you compiling and interfacing the Fortran module? [tag:f2py]? Or [tag:ctypes]? Or something else? A simple example ([mcve]) would be helpful. – Vladimir F Героям слава May 18 '17 at 10:18
  • Thanks for the suggestion - I have updated the question accordingly. – Amos Egel May 18 '17 at 12:56
  • I expect that your FORTRAN extension didn't link some of the required DLLs with it. Which explains why it did work after installing the correct MinGW version (which includes the missing DLLs). What is confusing is that the error message states that the DLL was not found, while in reality it couldn't find some other DLL that wasn't included. Did you use gfortran? Cause it is very difficult to link the required DLLs with a FORTRAN extension when using gfortran (somebody know how to do that officially?). – Mark Bakker Jul 24 '17 at 14:37
  • @Mark Thanks for the hint. Yes, I was using gfortran. Would you recommend to use a different compiler? – Amos Egel Jul 25 '17 at 15:06
  • I recon from your reply that you have not solved this problem? (If so, please post it here). Anyway, I ran into the same problem. There is no standard option in gfortran to link the required dlls or syslibs (Mac), at least not when I tried it. So you have to hack gfortran a bit to do that. Let me see if I can find back how I did that. Regarding your question: I suspect (or at least hope) that this may be easier with other compilers, but I am not sure. – Mark Bakker Jul 26 '17 at 15:40
  • No, I haven't solved the problem. Currently I am not using wheels but just ship a compiled .exe file for Win users and for Linux users, my package compiles the Fortran code to an executable the first time it is run by just calling gfortran from Python through the system command. I know this is not a clean way to do it, but it does have the advantage that I can ship the Fortran package (which is 3rd party) in its original form. – Amos Egel Jul 28 '17 at 07:33

2 Answers2

6

I recently ran into this issue by writing my own f2py building tool chain by compiling and linking all the components individually. The script was finding or installing required compilers automatically if they werent already found on the path. For cases where the gfortran tools werent on the path, but were present on the machine, I was able to inject the correct environment variables to os.environ and spawn compiler calls using Popen and the set of environment variables so that the pyd would compile. But outside of that python instance the environment variables were not correct for the pyd to run, I was getting the same DLL load failed error even on the same computer that compiled the pyds but which didnt have the correct paths setup.

So, since I'm compiling all steps separately, only using f2py to generate the f and c wrappers, I simply added -static -static-libgfortran -static-libgcc to my link step, and this causes the pyd to include the required libraries to run on those machines without the correct environment variables.

Achieving the same using numpy.distutils is possible (thanks to https://github.com/numpy/numpy/issues/3405):

from numpy.distutils.core import Extension, setup


if __name__ == "__main__":
    setup(
        name="this",
        ext_modules=[
            Extension("fortmod_nostatic",
                      ["src/code.f90"],
                      ),
            Extension("fortmod_withstatic",
                      ["src/code.f90"],
                      extra_link_args=["-static", "-static-libgfortran", "-static-libgcc"]
                      )
        ]
    )

I put the above in a file test.py and built with python test.py build_ext --inplace --compiler=mingw32 --fcompiler=gnu95 -f

For comparison there is a clear size difference. Inspecting the pyd's with dependency walker shows the nostatic one depends on libgfortran-4.dll whereas the extra flags generate a pyd that does not depend on this library. In my case after adding the static flags the machine without correct environment variables is able to run the pyds, and I suspect this case will be similar to yours since the dependency on libgfortran is removed.

Hope that helps! my first SO post..

rmar_
  • 86
  • 3
  • 7
  • Thanks a lot, this is really helpful. I can confirm that with the given extra_link_args and using the command `python setup.py build_ext --inplace --compiler=mingw32 --fcompiler=gnu95 -f`, the created pyd file does not depend on libgfortran anymore. However, it seems that running setup.py with any other command (develop, bdist_wheel, ...) does not result in a statically linked extension :-/ – Amos Egel Nov 30 '20 at 23:40
0

I can confirm that the steps described in rmar_'s answer create a pyd file that can be imported from a machine that has no MinGW installed. This is really helpful!

In order to create a binary wheel, I do the following:

  1. Follow the steps described in rmar_'s answer in order to generate statically linked pyd files in place.
  2. Run python setup.py bdist_wheel afterwards.

For the second step, I switch off the compilation of the pyd file (e.g., by removing the ext_modules keyword from the setup() call), because I want that the pyd file created in the first step is used (and not a newly created, which might be not statically linked).

I don't know if this makes sense at all, but it seems to work ...

Amos Egel
  • 937
  • 10
  • 24