In a setup.py involving Cython, if install_requires, then how can from library import something?

Question

This doesn't make sense to me. How can I use the setup.py to install Cython and then also use the setup.py to compile a library proxy?

import sys, imp, os, glob
from setuptools import setup
from Cython.Build import cythonize # this isn't installed yet

setup(
    name='mylib',
    version='1.0',
    package_dir={'mylib': 'mylib', 'mylib.tests': 'tests'},
    packages=['mylib', 'mylib.tests'],
    ext_modules = cythonize("mylib_proxy.pyx"), #how can we call cythonize here?
    install_requires=['cython'],
    test_suite='tests',
)

Later: python setup.py build

Traceback (most recent call last):
  File "setup.py", line 3, in <module>
    from Cython.Build import cythonize
ImportError: No module named Cython.Build

It's because cython isn't installed yet.

What's odd is that a great many projects are written this way. A quick github search reveals as much: https://github.com/search?utf8=%E2%9C%93&q=install_requires+cython&type=Code

Answer 1

As I understand it, this is where PEP 518 comes in - also see some clarifications by one of its authors.

The idea is that you add yet another file to your Python project / package: pyproject.toml. It is supposed to contain information on build environment dependencies (among other stuff, long term). pip (or just any other package manager) could look into this file and before running setup.py (or any other build script) install the required build environment. A pyproject.toml could therefore look like this:

[build-system]
requires = ["setuptools", "wheel", "Cython"]

It is a fairly recent development and, as of yet (January 2019), it is not finalized / approved by the Python community, though (limited) support was added to pip in May 2017 / the 10.0 release.

Answer 2

One solution to this is to not make Cython a build requirement, and instead distribute the Cython generated C files with your package. I'm sure there is a simpler example somewhere, but this is what pandas does - it conditionally imports Cython, and if not present can be built from the c files.

https://github.com/pandas-dev/pandas/blob/3ff845b4e81d4dde403c29908f5a9bbfe4a87788/setup.py#L433

Edit: The doc link from @danny has an easier to follow example. http://docs.cython.org/en/latest/src/reference/compilation.html#distributing-cython-modules

Answer 3

When you use setuptool, you should add cython to setup_requires (and also to install_requires if cython is used by installation), i.e.

# don't import cython, it isn't yet there
from setuptools import setup, Extension

# use Extension, rather than cythonize (it is not yet available)
cy_extension = Extension(name="mylib_proxy", sources=["mylib_proxy.pyx"])

setup(
    name='mylib',
    ...
    ext_modules = [cy_extension],
    setup_requires=["cython"],
    ...
)

Cython isn't imported (it is not yet available when setup.pystarts), but setuptools.Extension is used instead of cythonize to add cython-extension to the setup.

It should work now. The reason: setuptools will try to import cython, after setup_requires are fulfilled:

...
try:
    # Attempt to use Cython for building extensions, if available
    from Cython.Distutils.build_ext import build_ext as _build_ext
    # Additionally, assert that the compiler module will load
    # also. Ref #1229.
    __import__('Cython.Compiler.Main')
except ImportError:
    _build_ext = _du_build_ext
...

It becomes more complicated, if your Cython-extension uses numpy, but also this is possible - see this SO post.

Answer 4

It doesn't make sense in general. It is, as you suspect, an attempt to use something that (possibly) has yet to be installed. If tested on a system that already has the dependency installed, you might not notice this defect. But run it on a system where your dependency is absent, and you will certainly notice.

There is another setup() keyword argument, setup_requires, that can appear to be parallel in form and use to install_requires, but this is an illusion. Whereas install_requires triggers a lovely ballet of automatic installation in environments that lack the dependencies it names, setup_requires is more documentation than automation. It won't auto-install, and certainly not magically jump back in time to auto-install modules that have already been called for in import statements.

There's more on this at the setuptools docs, but the quick answer is that you're right to be confused by a module that is trying to auto-install its own setup pre-requisites.

For a practical workaround, try installing cython separately, and then run this setup. While it won't fix the metaphysical illusions of this setup script, it will resolve the requirements and let you move on.