How to Bootstrap numpy installation in setup.py

Question

I have a project which has a C extension which requires numpy. Ideally, I'd like whoever downloads my project to just be able to run python setup.py install or use one call to pip. The problem I have is that in my setup.py I need to import numpy to get the location of the headers, but I'd like numpy to be just a regular requirement in install_requires so that it will automatically be downloaded from the Python Package Index.

Here is a sample of what I'm trying to do:

from setuptools import setup, Extension
import numpy as np

ext_modules = [Extension('vme', ['vme.c'], extra_link_args=['-lvme'],
                         include_dirs=[np.get_include()])]

setup(name='vme',
      version='0.1',
      description='Module for communicating over VME with CAEN digitizers.',
      ext_modules=ext_modules,
      install_requires=['numpy','pyzmq', 'Sphinx'])

Obviously, I can't import numpy at the top before it's installed. I've seen a setup_requires argument passed to setup() but can't find any documentation on what it is for.

Is this possible?

score 30 · Answer 1 · edited May 23 '17 at 10:30

30

The following works at least with numpy1.8 and python{2.6,2.7,3.3}:

from setuptools import setup
from setuptools.command.build_ext import build_ext as _build_ext

class build_ext(_build_ext):
    def finalize_options(self):
        _build_ext.finalize_options(self)
        # Prevent numpy from thinking it is still in its setup process:
        __builtins__.__NUMPY_SETUP__ = False
        import numpy
        self.include_dirs.append(numpy.get_include())

setup(
    ...
    cmdclass={'build_ext':build_ext},
    setup_requires=['numpy'],
    ...
)

For a small explanation, see why it fails without the "hack", see this answer.

Note, that using setup_requires has a subtle downside: numpy will not only be compiled before building extensions, but also when doing python setup.py --help, for example. To avoid this, you could check for command line options, like suggested in https://github.com/scipy/scipy/blob/master/setup.py#L205, but on the other hand I don't really think it's worth the effort.

edited May 23 '17 at 10:30

Community

1
1

answered Feb 07 '14 at 07:24

coldfix

6,604
3
40
50

I wonder how can I do this and use `Cython.distutils.build_ext` at the same time... – Capi Etheriel Oct 14 '14 at 19:36
It might just work the same way by replacing `setuptools.command` by `Cython.distutils` in the example above. If not: I was using `cythonize()` instead of the `Cython.distutils.build_ext` which is recommended anyway. See [here](https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing#example) and [here](http://docs.cython.org/src/reference/compilation.html#compiling-with-distutils) – coldfix Oct 15 '14 at 17:33
@coldfix yes, but then I'll need to have Cython installed before running setup.py. Bootstraping setup.py seems to be a pain :P – Capi Etheriel Nov 03 '14 at 12:56

score 6 · Answer 2 · edited Feb 27 '23 at 14:04

6

I found a very easy solution:

Or you can stick to https://github.com/pypa/pip/issues/5761. Here you install cython and numpy using setuptools.dist before actual setup:

from setuptools import dist
dist.Distribution().fetch_build_eggs(['Cython>=0.15.1', 'numpy>=1.10'])

Works well for me!

edited Feb 27 '23 at 14:04

vvvvv

25,404
19
49
81

answered Mar 18 '20 at 13:35

floschl

81
1
4

Wow, this is the simplest answer and it is working for me as well! It also works with readthedocs website, in the case, one needs to install the package itself. – Mikhail Genkin Aug 27 '20 at 19:55
Unfortunately with modern Python this is deprecated. – Pux Jul 30 '23 at 15:25

score 4 · Answer 3 · answered Nov 12 '13 at 07:55

This is a fundamental problem with packages that need to use numpy (for distutils or get_include). I do not know of a way to "boot-strap" it using pip or easy-install.

However, it is easy to make a conda package for your module and provide the list of dependencies so that someone can just do a conda install pkg-name which will download and install everything needed.

Conda is available in Anaconda or in Miniconda (python + just conda).

See this website: http://docs.continuum.io/conda/index.html or this slide-deck for more information: https://speakerdeck.com/teoliphant/packaging-and-deployment-with-conda

score 4 · Answer 4 · answered Jul 04 '20 at 03:09

The key is to defer importing numpy until after it has been installed. A trick I learned from this pybind11 example is to import numpy in the __str__ method of a helper class (get_numpy_include below).

from setuptools import setup, Extension

class get_numpy_include(object):
    """Defer numpy.get_include() until after numpy is installed."""

    def __str__(self):
        import numpy
        return numpy.get_include()


ext_modules = [Extension('vme', ['vme.c'], extra_link_args=['-lvme'],
                         include_dirs=[get_numpy_include()])]

setup(name='vme',
      version='0.1',
      description='Module for communicating over VME with CAEN digitizers.',
      ext_modules=ext_modules,
      install_requires=['numpy','pyzmq', 'Sphinx'])

score 3 · Answer 5 · answered Nov 13 '13 at 13:03

3

To get pip to work, you can do similarly as Scipy: https://github.com/scipy/scipy/blob/master/setup.py#L205

Namely, the egg_info command needs to be passed to standard setuptools/distutils, but other commands can use numpy.distutils.

answered Nov 13 '13 at 13:03

pv.

33,875
8
55
49

Links should always use the sha (you can access from "copy permalink" in GitHub) in case the code changes and the line numbers change. It looks like that's happened here. From the code mentioned, I'm guessing this is the line https://github.com/scipy/scipy/blob/5a1072d651ad9f64f34da4f4532c31c859a82911/setup.py#L205 – Keegan Jun 24 '21 at 19:12

score 3 · Answer 6 · answered Jun 18 '21 at 15:46

This should now (since 2018-ish) be solved by adding numpy as a buildsystem dependency in pyproject.toml, so that pip install makes numpy available before it runs setup.py.

The pyproject.toml file should also specify that you're using Setuptools to build the project. It should look something like this:

[build-system]
requires = ["setuptools", "wheel", "numpy"]
build-backend = "setuptools.build_meta"

See Setuptools' Build System Support docs for more details.

This doesn't cover many other uses of setup.py other than install, but as those are mainly for you (and other developers of your project), so an error message saying to install numpy might work.

Kamil Sindi · Answer 7 · 2017-01-10T18:34:25.267

Perhaps a more practical solution is to just require numpy to be installed beforehand and import numpy inside a function scope. @coldfix solution works but compiling numpy takes forever. Much faster to pip install it first as a wheels package, especially now that we have wheels for most systems thanks to efforts like manylinux.

from __future__ import print_function

import sys
import textwrap
import pkg_resources

from setuptools import setup, Extension


def is_installed(requirement):
    try:
        pkg_resources.require(requirement)
    except pkg_resources.ResolutionError:
        return False
    else:
        return True

if not is_installed('numpy>=1.11.0'):
    print(textwrap.dedent("""
            Error: numpy needs to be installed first. You can install it via:

            $ pip install numpy
            """), file=sys.stderr)
    exit(1)

def ext_modules():
    import numpy as np

    some_extention = Extension(..., include_dirs=[np.get_include()])

    return [some_extention]

setup(
    ext_modules=ext_modules(),
)

score 1 · Answer 8 · answered Jan 11 '19 at 20:02

@coldfix's solution doesn't work for Cython-extensions, if Cython isn't pre-installed on the target-machine, as it fails with the error

error: unknown file type '.pyx' (from 'xxxxx/yyyyyy.pyx')

The reason for the failure is the premature import of setuptools.command.build_ext, because when imported, it tries to use Cython's build_ext-functionality:

try:
    # Attempt to use Cython for building extensions, if available
    from Cython.Distutils.build_ext import build_ext as _build_ext
    # Additionally, assert that the compiler module will load
    # also. Ref #1229.
    __import__('Cython.Compiler.Main')
except ImportError:
_build_ext = _du_build_ext

And normally setuptools is successful, because the import happens after setup_requirements are fulfilled. However by importing it already in setup.py, only fall back solution can be used, which doesn't know any about Cython.

One possibility to bootstrap Cython alongside with numpy, would be to postpone the import of setuptools.command.build_ext with help of an indirection/proxy:

# factory function
def my_build_ext(pars):
     # import delayed:
     from setuptools.command.build_ext import build_ext as _build_ext#

     # include_dirs adjusted: 
     class build_ext(_build_ext):
         def finalize_options(self):
             _build_ext.finalize_options(self)
             # Prevent numpy from thinking it is still in its setup process:
             __builtins__.__NUMPY_SETUP__ = False
             import numpy
             self.include_dirs.append(numpy.get_include())

    #object returned:
    return build_ext(pars)

...
setup(
    ...
    cmdclass={'build_ext' : my_build_ext},
    ...
)

There are other possibilities, discussed for example in this SO-question.

score 0 · Answer 9 · answered Jul 29 '22 at 23:20

0

You can simply add numpy into your pyproject.toml file. This works for me.

[build-system] 
requires = [ 
    "setuptools>=42",
    "wheel", 
    "Cython", 
    "numpy" 
] 
build-backend = "setuptools.build_meta"

answered Jul 29 '22 at 23:20

Ethan Steere

11
1

score 0 · Answer 10 · answered Apr 21 '23 at 17:17

0

I don't have enough reputation to comment on threads above, it looks like __NUMPY_SETUP__ is not required since numpy 1.13+

https://numpy.org/doc/1.13/release.html

#7956: BLD: remove NUMPY_SETUP from builtins at end of setup.py

answered Apr 21 '23 at 17:17

Mateusz Kraiński

75
7

How to Bootstrap numpy installation in setup.py

10 Answers10

Linked