1

Given is a (set of) Python3 packages that is to be deployed in different scenarios either cythonized or as original scripts; the source are pure Python3 sources. Preferably, I would like to use the same setup.py, if possible.

use case in-place include .py modules cythonized .so modules
1. development pip3 install -e . yes yes
2. "unoptimized" install pip3 install . yes
3. cythonized install pip3 install . --install-option="cythonize" no (except __init__.py) yes
4. build (binary) wheel python3 setup.py bdist_wheel no (except __init__.py) yes

So far, I succeeded in building a binary distribution wheel with only the cythonized .so shared library and without the original .py module files, following Package only binary compiled .so files of a python library compiled with Cython . This covers use case #4 and is handled by class build_py.

However, I would also cover #1, #2 and maybe #3; #3 might be better tackled by separately building the bdist_wheel and then installing this, of not otherwise possible in a single step.

# https://stackoverflow.com/a/56043918
from setuptools.command.build_py import build_py as build_py_orig
try:
    from Cython.Build import cythonize
except:
    cythonize = None

from setuptools.command.install import install as install_orig

# https://stackoverflow.com/a/56043918
extensions = [
    Extension('spam.*', ['spam/**/*.py'],
              extra_compile_args=["-O3", "-Wall"]),
]

cython_excludes = ['spam/**/__init__.py']

def not_cythonized(tup):
    (package, module, filepath) = tup
    return any(
        fnmatch.fnmatchcase(filepath, pat=pattern) for pattern in cython_excludes
    ) or not any(
        fnmatch.fnmatchcase(filepath, pat=pattern)
        for ext in extensions
        for pattern in ext.sources
    )


class build_py(build_py_orig):
    def find_modules(self):
        modules = super().find_modules()
        return list(filter(not_cythonized, modules))

    def find_package_modules(self, package, package_dir):
        modules = super().find_package_modules(package, package_dir)
        return list(filter(not_cythonized, modules))


class install(install_orig):
    def finalize_options(self):
        super().finalize_options()
        self.distribution.ext_modules = None

setup(
    name='spam',
    packages=find_packages(),
    ext_modules=cythonize(
        extensions,
        exclude=cython_excludes,
        compiler_directives={
            "language_level": 3,
            "always_allow_keywords": True,
        },
        build_dir="build",  # needs to be explicitly set, otherwise pollutes package sources
    ) if cythonize is not None else [],
    cmdclass={
        'build_py': build_py,
        'install': install,
    },
    include_package_data=True,
    install_requires=[...]
)

The problems I'm facing here:

  1. for use cases #1 and #2 I don't want to cythonize, so ext_modules= should not be specified/set.

    1. What is a sensible way to handle ext_modules= in this situation? I find it hard to detect the requested operation (install, install -e, develop) before calling setup(), so would it be better to inherit and override the install and develop classes?

    2. If the latter, is it possible and allowed to clear the ext_modules and how do I avoid prematurely evaluating cythonize(...)?

  2. in use case #2 with the above code pip3 decides to build an egg which unfortunately includes the .so's. Might this be due to cythonize(...) getting evaluated in any case? can I avoid building the egg or how do I prevent the egg build process from including the shared libs?

  3. this currently includes both the sources (which I don't want to be include) as well as the cythonized modules: how can I prevent the install class from installing most of the source modules, yet installing the __init__.pys?

TheDiveO
  • 2,183
  • 2
  • 19
  • 38

0 Answers0