setup_requires with Cython?

Question

I'm creating a setup.py file for a project with some Cython extension modules.

I've already gotten this to work:

from setuptools import setup, Extension
from Cython.Build import cythonize

setup(
    name=...,
    ...,
    ext_modules=cythonize([ ... ]),
)

This installs fine. However, this assumes Cython is installed. What if it's not installed? I understand this is what the setup_requires parameter is for:

from setuptools import setup, Extension
from Cython.Build import cythonize

setup(
    name=...,
    ...,
    setup_requires=['Cython'],
    ...,
    ext_modules=cythonize([ ... ]),
)

However, if Cython isn't already installed, this will of course fail:

$ python setup.py install
Traceback (most recent call last):
  File "setup.py", line 2, in <module>
    from Cython.Build import cythonize
ImportError: No module named Cython.Build

What's the proper way to do this? I need to somehow import Cython only after the setup_requires step runs, but I need Cython in order to specify the ext_modules values.

score 33 · Answer 1 · edited Feb 23 '23 at 19:29

33

Starting from 18.0 release of setuptools (released on 2015-06-23) it is possible to specify Cython in setup_requires and pass *.pyx modules sources for regular setuptools.Extension:

from setuptools import setup, Extension

setup(
    # ...
    setup_requires=[
        # Setuptools 18.0 properly handles Cython extensions.
        'setuptools>=18.0',
        'cython',
    ],
    ext_modules=[
        Extension(
            'mylib',
            sources=['src/mylib.pyx'],
        ),
    ],
)

edited Feb 23 '23 at 19:29

vvvvv

25,404
19
49
81

answered Jun 27 '16 at 14:56

rutsky

3,900
2
31
30

Great answer, thank you! Would you prefer this approach next to the approach were Cython files are compiled in advance and .c files are used in setup.py, which means that Cython is not needed to build the package? – Martinsos Feb 16 '17 at 12:39
1

@Martinsos assuming that Cython is installable on all target platforms using setuptools (it should, but there are always exceptions), using specified approach removes hassle of generating those .c files during library deployment without loosing anything, so yes, I would recommend this approach. The only exception if this process doesn't work, e.g. if you target system with very old setuptools, which I wouldn't by default. – rutsky Feb 16 '17 at 18:33
3

Are you sure it works? I tried with Python 3.5, setuptools 39.0.1. If cython is not installed, it gives error: unknown file type '.pyx' – Labo Apr 05 '18 at 14:05
@Labo I assume you opened this bug report for mentioned issue? https://github.com/pypa/setuptools/issues/1317 – rutsky Apr 19 '18 at 21:55
@rutsky indeed ;) – Labo Apr 20 '18 at 11:41
1

This is very useful. In this setup, without importing cython, do you know how we could set cython compiler options, as we would do with: ``` from Cython.Compiler import Options Options.annotate = True ``` – Py_Dream Nov 09 '19 at 18:53

mobiusklein · Accepted Answer · 2016-05-26T23:52:01.387

4

You must wrap the from Cython.Build import cythonize in a try-except, and in the except, define cythonize as a dummy function. This way the script can be loaded without failing with an ImportError.

Then later when the setup_requires argument is handled, Cython will be installed and the setup script will be re-executed. Since at that point Cython is installed, you'll be able to successfully import cythonize

try:
    from Cython.Build import cythonize
except ImportError:
     def cythonize(*args, **kwargs):
         from Cython.Build import cythonize
         return cythonize(*args, **kwargs)

EDIT

As noted in comments, after setuptools deals with missing dependencies, it won't re-load Cython. I hadn't thought of it before, but you could also try a late-binding approach to stubbing out cythonize

edited May 26 '16 at 23:52

answered May 26 '16 at 21:37

mobiusklein

1,403
9
12

That's somewhat hacky, and yet, pretty cool! Didn't know the script is re-executed, that makes a lot of sense. – Claudiu May 26 '16 at 21:51
Hmm, so if I install without Cython having been installed first, the setup works (it installs Cython) but the extension modules aren't installed (can't import them from scripts). Any ideas of what may be happening? – Claudiu May 26 '16 at 21:56
Curious. Ah well it must not be truly re-executed. I usually don't require `Cython` for installation, just development. Instead of making `Extension` instances with `cythonize`, just create them directly on the `Cython` generated .c files. Checkout this example https://github.com/mobiusklein/brainpy/blob/master/setup.py – mobiusklein May 26 '16 at 22:20
I should clarify that in that case you'll also need to make sure to include .h, .pyx, pxd, and .c files in the MANIFEST file for your distribution. – mobiusklein May 26 '16 at 22:32
Oh hmm, that's not a bad idea. I'll keep that floating around in the ol' noggin for a bit. – Claudiu May 26 '16 at 23:35
How is one supposed to pass `gdb_debug=True` to `cythonize` if the function is not used? – astrojuanlu Jun 30 '19 at 16:45

score 4 · Answer 3 · answered Jul 14 '20 at 09:19

There seems to be a third way of having build dependencies installed before executing the actual setup.py described here (requires pip):

https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#basic-setup-

In essence:

Create the file pyproject.toml with the following contents:

[build-system]
requires = ["setuptools", "wheel", "Cython"]

Use pip install -e . for setting things up

setup_requires with Cython?

3 Answers3

Linked