53

I am trying to make a python package distribution for some popular Fortran codes in my field. I want it to use the most standard approach with a setup.py file. The related qustion was helpful for learning how to wrap Fortran extensions.

When using this approach, I noticed some confusing behavior when mixing setuptools and numpy.distutils. Is it bad practice to mix the two? As of 2015, it seems preferable to use setuptools as much as possible.

However, I would like to build Fortran extensions in a way that is compatible with numpy. So I would like to import from numpy.distutils to get Extension and setup.

I'm using the following basic approach:

from setuptools.command.develop import develop
from numpy.distutils.core import Extension, setup

ext_modules=[Extension("my_package.fortran_mod", sources=['src/fortran_mod.f'])]

class MyDevelop(develop):
  def run(self):
    my_script()
    develop.run(self)

setup(
  ...
  ext_modules=ext_modules,
  cmdclass={'develop':MyDevelop})

This seems to work but I have questions.

  1. Is it generally good practice to mix setuptools and numpy.distribute?
  2. Does the order I import them matter? Should I always import setuptools first?
  3. Is there an official up-to-date tutorial for packaging extensions to numpy? Perhaps even one with some discussion Fortran extensions?

Some links

https://www.youtube.com/watch?v=R4yB-8tB0J0

http://www.fortran90.org/src/best-practices.html#interfacing-with-python

Vladimir F Героям слава
  • 57,977
  • 4
  • 76
  • 119
Will Martin
  • 993
  • 6
  • 18
  • 5
    I'm not an expert in wrapping Fortran code - but I'm pretty experience with Cython. [This talk](https://www.youtube.com/watch?v=R4yB-8tB0J0) from Scipy 2015 has a really nice section on successfully wrapping Fortran code - I suggest you take a look! – BeRecursive Jul 20 '15 at 08:51
  • 1
    Thanks. I really appreciate your comment. That is new information for me and seems useful. Could you list a few of the main advantages of writing the wrapper layer in Cython over standard Python. Is it just faster? or is there some improved generality? He stressed the `nogil` at one point. Maybe this would be advantageous if the code I'm wrapping uses MPI? – Will Martin Jul 20 '15 at 13:19
  • 2
    What do you mean by standard Python? ``nogil`` is very important if you want to be able to delegate down to native code and make use of threading/multi-process. Your MPI example is a perfect time to release the GIL! – BeRecursive Jul 20 '15 at 20:16
  • 1
    "Standard" isn't the right word I guess. A package is either pure Python or it has extensions, and I'm already using extensions. I have only used Fortran extensions so far. I will need to wrap Fortran-MPI code soon, so if Cython lets me turn off the `gil` that is fantastic. Thanks for sharing the video and bringing up this point. – Will Martin Jul 20 '15 at 20:45
  • 1
    @BeRecursive How does the GIL have anything common with MPI? MPI has separate processes so the lock should not come into play. Threads are different though. – Vladimir F Героям слава Jul 21 '15 at 17:01
  • Thanks for clarifying, Vladimir. – Will Martin Jul 21 '15 at 20:02
  • 2
    Compared to f2py, Cython would allow you to use [iso_c_binding](https://gcc.gnu.org/onlinedocs/gfortran/ISO_005fC_005fBINDING.html) in the Fortran code.Because f2py is mainly build for Fortran77 code, it lacks a lot of great Fortran90+ features, such as dynamic memory allocation. With Cython you have a much greater flexibility because your Fortran and Cython code can speak (almost natively) to eachother using iso_c_binding. – Max Graser Nov 09 '15 at 14:26
  • 2
    Following some of the earlier comments I would like to contribute two more options to your considerations. If you use "old" Fortran, then you could directly access your Fortran functions using `ctypes` (just take care of the function-names which, depending on your compiler, get a trailing underscore). Recently I have become a fan of [pybind11](https://github.com/pybind/pybind11) which allows you to write the entire Python module from C++, with the bonus advantage that the ancient Fortran code gets a modern front-end. – Tom de Geus Mar 31 '17 at 07:47
  • 1
    It appears that Cython provides a very flexible way to wrap FORTRAN programs. Here is a working example of how to follow the Cython (with iso_c_binding) route: https://stackoverflow.com/questions/22404060/fortran-cython-workflow . – Will Martin Jul 10 '17 at 12:11
  • 2
    @Will Before you undertake your Fortran conversions are you positive they aren't already implemented in the SciPy low level Fortran interfaces? https://github.com/scipy/scipy/blob/master/scipy/linalg/blas.py also lapack.py you'd be hard pressed to write anything faster (I've tried with MKL and can only shave maybe microseconds (I.e. not statistically significant improvements)) Also have a look there they have some prewitten Cython test programs (dgemm does have a bug you can resolve by searching on a post of mine SciPy dgemm interface not working on mx1 1xn vectors) – Matt Jul 30 '17 at 19:21

1 Answers1

3

This seems to work but I have questions.

  1. Is it generally good practice to mix setuptools and numpy.distribute?
  2. Does the order I import them matter? Should I always import setuptools first?
  3. Is there an official up-to-date tutorial for packaging extensions to numpy? Perhaps even one with some discussion Fortran extensions?

  1. You should not need to use numpy.distribute anymore.

  2. ^^ Not necessary

  3. Particularly for wrapping fortran code with numpy, there is the popular f2py. However I personally find the necessary code annotations redundant, because good fortran code contains all necessary information.

(warning personal project plug below)

Recently released is the cleaner fmodpy, which automatically generates all necessary wrapper code in an understandable and clean interface. It supports pre-Fortran90, but is best suited for Fortran90 and later. It could be used to generate a clean distribution along with the python interface of code (presuming users have gfortran installed).

Thomas Lux
  • 1,048
  • 9
  • 16
  • 1
    The personal project plug is not problem, but this answer is. Could you please reference the questions you are answering int the 1,2,3 above so that this *answer* can be actually read as is? – Stephen Rauch Dec 13 '17 at 03:28