Given is a (set of) Python3 packages that is to be deployed in different scenarios either cythonized or as original scripts; the source are pure Python3 sources. Preferably, I would like to use the same setup.py
, if possible.
use case | in-place | include .py modules | cythonized .so modules |
---|---|---|---|
1. development pip3 install -e . |
yes | yes | |
2. "unoptimized" install pip3 install . |
yes | ||
3. cythonized install pip3 install . --install-option="cythonize" |
no (except __init__.py ) |
yes | |
4. build (binary) wheel python3 setup.py bdist_wheel |
no (except __init__.py ) |
yes |
So far, I succeeded in building a binary distribution wheel with only the cythonized .so shared library and without the original .py module files, following Package only binary compiled .so files of a python library compiled with Cython
. This covers use case #4 and is handled by class build_py
.
However, I would also cover #1, #2 and maybe #3; #3 might be better tackled by separately building the bdist_wheel and then installing this, of not otherwise possible in a single step.
# https://stackoverflow.com/a/56043918
from setuptools.command.build_py import build_py as build_py_orig
try:
from Cython.Build import cythonize
except:
cythonize = None
from setuptools.command.install import install as install_orig
# https://stackoverflow.com/a/56043918
extensions = [
Extension('spam.*', ['spam/**/*.py'],
extra_compile_args=["-O3", "-Wall"]),
]
cython_excludes = ['spam/**/__init__.py']
def not_cythonized(tup):
(package, module, filepath) = tup
return any(
fnmatch.fnmatchcase(filepath, pat=pattern) for pattern in cython_excludes
) or not any(
fnmatch.fnmatchcase(filepath, pat=pattern)
for ext in extensions
for pattern in ext.sources
)
class build_py(build_py_orig):
def find_modules(self):
modules = super().find_modules()
return list(filter(not_cythonized, modules))
def find_package_modules(self, package, package_dir):
modules = super().find_package_modules(package, package_dir)
return list(filter(not_cythonized, modules))
class install(install_orig):
def finalize_options(self):
super().finalize_options()
self.distribution.ext_modules = None
setup(
name='spam',
packages=find_packages(),
ext_modules=cythonize(
extensions,
exclude=cython_excludes,
compiler_directives={
"language_level": 3,
"always_allow_keywords": True,
},
build_dir="build", # needs to be explicitly set, otherwise pollutes package sources
) if cythonize is not None else [],
cmdclass={
'build_py': build_py,
'install': install,
},
include_package_data=True,
install_requires=[...]
)
The problems I'm facing here:
for use cases #1 and #2 I don't want to cythonize, so
ext_modules=
should not be specified/set.What is a sensible way to handle
ext_modules=
in this situation? I find it hard to detect the requested operation (install
,install -e
,develop
) before callingsetup()
, so would it be better to inherit and override theinstall
anddevelop
classes?If the latter, is it possible and allowed to clear the
ext_modules
and how do I avoid prematurely evaluatingcythonize(...)
?
in use case #2 with the above code pip3 decides to build an egg which unfortunately includes the
.so
's. Might this be due tocythonize(...)
getting evaluated in any case? can I avoid building the egg or how do I prevent the egg build process from including the shared libs?this currently includes both the sources (which I don't want to be include) as well as the cythonized modules: how can I prevent the
install
class from installing most of the source modules, yet installing the__init__.py
s?