Make distutils look for numpy header files in the correct place

Question

In my installation, numpy's arrayobject.h is located at …/site-packages/numpy/core/include/numpy/arrayobject.h. I wrote a trivial Cython script that uses numpy:

cimport numpy as np

def say_hello_to(name):
    print("Hello %s!" % name)

I also have the following distutils setup.py (copied from the Cython user guide):

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

ext_modules = [Extension("hello", ["hello.pyx"])]

setup(
  name = 'Hello world app',
  cmdclass = {'build_ext': build_ext},
  ext_modules = ext_modules
)

When I try to build with python setup.py build_ext --inplace, Cython tries to do the following:

gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd \
-fno-common -dynamic -DNDEBUG -g -Os -Wall -Wstrict-prototypes -DMACOSX \
-I/usr/include/ffi -DENABLE_DTRACE -arch i386 -arch ppc -pipe \
-I/System/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 \
-c hello.c -o build/temp.macosx-10.5-i386-2.5/hello.o

Predictably, this fails to find arrayobject.h. How can I make distutils use the correct location of numpy include files (without making the user define $CFLAGS)?

score 73 · Accepted Answer · edited Apr 16 '20 at 19:21

73

Use numpy.get_include():

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import numpy as np                           # <---- New line

ext_modules = [Extension("hello", ["hello.pyx"],
                                  include_dirs=[get_numpy_include()])]   # <---- New argument

setup(
  name = 'Hello world app',
  cmdclass = {'build_ext': build_ext},       
  ext_modules = ext_modules
)

edited Apr 16 '20 at 19:21

ead

32,758
6
90
153

answered Mar 04 '10 at 14:19

Vebjorn Ljosa

17,438
13
70
88

2

I have the same problem when using the `%%cython` magic in an ipython notebook.. I wonder if there is an easy fix for that as well – pbreach Sep 14 '14 at 06:26
2

For anyone looking at this as an example for non-`numpy` related work, the definition of `numpy.get_include` is [here](https://github.com/numpy/numpy/blob/56678fe56dce97871bb49febf0b2c0206541eada/numpy/lib/utils.py#L18). – 0 _ Jul 07 '15 at 04:30
1

im not sure if i implemented it correctly but it seems like this renders `setup.py` useless for remote installations. You get an error ` ModuleNotFoundError: No module named 'numpy'` – Erik K May 17 '19 at 16:42
will try to make it work with pipenv, works with `pip install` thanks – Erik K May 19 '19 at 18:46
pip 19 and above breaks all of this again due to enforcement of PEP517/518 – Erik K Jun 12 '19 at 16:12
1

this works, but do not work in releasing it to pypi since I do not have numpy package installed! – Jingpeng Wu Aug 15 '19 at 15:07
6

Should `get_numpy_include()` not be `np.get_include()`? – Demitri Apr 26 '20 at 09:34
distutils is deprecated – user3673 May 04 '22 at 21:06

score 32 · Answer 2 · answered Feb 10 '17 at 15:32

The answer given by @vebjorn-ljosa is correct, but it causes problems when used in conjunction with install_requires=['numpy']. In this situation, your setup.py needs to import numpy, which will cause an error if you try to pip install your project without running pip install numpy first.

If your project depends on numpy, and you want numpy to be installed automatically as a dependency, you need to set include_dirs only when your extensions are actually being built. You can do this by subclassing build_ext:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

class CustomBuildExtCommand(build_ext):
    """build_ext command for use when numpy headers are needed."""
    def run(self):

        # Import numpy here, only when headers are needed
        import numpy

        # Add numpy headers to include_dirs
        self.include_dirs.append(numpy.get_include())

        # Call original build_ext command
        build_ext.run(self)

ext_modules = [Extension("hello", ["hello.pyx"])]

setup(
  name = 'Hello world app',
  cmdclass = {'build_ext': CustomBuildExtCommand},
  install_requires=['numpy'],
  ext_modules = ext_modules
)

And you can use a similar trick to add cython as an automatically installed dependency:

from distutils.core import setup
from distutils.extension import Extension

try:
    from Cython.setuptools import build_ext
except:
    # If we couldn't import Cython, use the normal setuptools
    # and look for a pre-compiled .c file instead of a .pyx file
    from setuptools.command.build_ext import build_ext
    ext_modules = [Extension("hello", ["hello.c"])]
else:
    # If we successfully imported Cython, look for a .pyx file
    ext_modules = [Extension("hello", ["hello.pyx"])]

class CustomBuildExtCommand(build_ext):
    """build_ext command for use when numpy headers are needed."""
    def run(self):

        # Import numpy here, only when headers are needed
        import numpy

        # Add numpy headers to include_dirs
        self.include_dirs.append(numpy.get_include())

        # Call original build_ext command
        build_ext.run(self)

setup(
  name = 'Hello world app',
  cmdclass = {'build_ext': CustomBuildExtCommand},
  install_requires=['cython', 'numpy'],
  ext_modules = ext_modules
)

Note: these approaches only work with pip install .. They won't work for python setup.py install or python setup.py develop as in these commands cause dependencies to be installed after your project, rather than before.

score 0 · Answer 3 · answered Feb 15 '19 at 21:01

For anyone not using Cython, a slight modification of R_Beagrie's solution without that dependency is if you simply import build_ext from distutils.command.build_ext instead of Cython.

from distutils.core import setup
from distutils.extension import Extension
from distutils.command.build_ext import build_ext

class CustomBuildExtCommand(build_ext):
    """build_ext command for use when numpy headers are needed."""
    def run(self):

        # Import numpy here, only when headers are needed
        import numpy

        # Add numpy headers to include_dirs
        self.include_dirs.append(numpy.get_include())

        # Call original build_ext command
        build_ext.run(self)

ext_modules = [Extension("hello", ["hello.c"])]

setup(
  name = 'Hello world app',
  cmdclass = {'build_ext': CustomBuildExtCommand},
  install_requires=['numpy'],
  ext_modules = ext_modules
)

Make distutils look for numpy header files in the correct place

3 Answers3

Linked