1

I am building a Python package after compilation using Cython and cythonize.

Here is an example of how my setup.py looks like:

from setuptools import setup, find_packages, Extension
from Cython.Build import cythonize


setup(
    name="My_Pkg",
    version="0.0",
    ext_modules=cythonize('lib/my_pkg/my_mod.py'),
)

And here is and example of how my project directory tree looks like:

.
├── lib
│   └── my_pkg
│       ├── __init__.py
│       └── my_mod.py
└── setup.py

When executing setuptools, everithing works fine:

MACOSX_DEPLOYMENT_TARGET=10.15.1 python setup.py build_ext bdist_wheel

My modules are correctly converted to '.c' and compiled to '.so' in the 'build' directory. When everything is completed, wheel is created in 'dist':

.
├── My_Pkg.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   └── top_level.txt
├── build
│   ├── bdist.macosx-10.7-x86_64
│   ├── lib.macosx-10.7-x86_64-3.7
│   │   └── my_pkg
│   │       └── my_mod.cpython-37m-darwin.so
│   └── temp.macosx-10.7-x86_64-3.7
│       └── lib
│           └── my_pkg
│               └── my_mod.o
├── dist
│   └── My_Pkg-0.0-cp37-cp37m-macosx_10_7_x86_64.whl
├── lib
│   └── my_pkg
│       ├── __init__.py
│       ├── my_mod.c
│       └── my_mod.py
└── setup.py

My problem is the I'd like the '__init__.py' to be copied in the 'build' as well (without being compiled) during the setuptools build phase. I also would like it to be packaged in the generated wheel as a part of the distribution.

Any idea?

Adriano
  • 487
  • 4
  • 8
  • Thanks to the help form @DavidW and https://stackoverflow.com/questions/50506072/exclude-single-source-file-from-python-bdist-egg-or-bdist-wheel I found a solution matching my requirements, I'll edit this post to add my new setup.py – Adriano Nov 22 '19 at 14:49
  • 1
    Glad you got it working. It's better to add your solution as an answer rather than edit it into your question (it's totally fine to answer your own questions) – DavidW Nov 22 '19 at 20:15

2 Answers2

1

To get this to work I had to add two keyword arguments to setup in setup.py:

packages=["my_pkg"],
package_dir={'': 'lib'},do two things:

(you could probably also use find_packages. This is how you tell setuptools about the Python packages it should be installing. Specifically we tell it that we have a package called "my_pkg" and it should look in lib for them. The best place to read about this is the distutils documentation (setuptools builds on distutils but mostly documents the changes).

Note that this doesn't include the .py files in the "build" folder. This is purely a temporary working space so it doesn't make sense to pointlessly copy files to it. You shouldn't be trying to import modules stashed in the "build" folder They are included in the wheel, which is the important point for being able to distribute them.

DavidW
  • 29,336
  • 6
  • 55
  • 86
  • I tried to your suggestions but this create a wrong wheel. The files get installed in '...site-packages/lib/my_pkg' instead of simply being installed in '...site-packages/my_pkg' as expected – Adriano Nov 22 '19 at 13:32
  • See edit - it's an extra arg to distutils (remove the `__init__.py` file from lib from the previous version) – DavidW Nov 22 '19 at 13:45
  • With the edited proposal, also 'my_mod.py' get copied to 'build' and also distributed in the wheel. – Adriano Nov 22 '19 at 13:53
  • I don't think that's a big problem - the .so file will be used in preference (if you're not convinced you can check this by importing the module then looking at the `__file__` attribute). If you really want to fix it though look at the distutils documentation I linked - I think you can specify individual modules instead – DavidW Nov 22 '19 at 14:01
  • I am convinced the dynamic lib will be imported if both are available but I'd like not to include them in the wheel. – Adriano Nov 22 '19 at 14:06
0

Thanks to the help form comments and this other post I came out with a solution using the following 'setup.py':

from setuptools import setup
from setuptools.command.build_py import build_py as build_py_orig
from Cython.Build import cythonize


class build_py(build_py_orig):

    def find_package_modules(self, package, package_dir):
        modules = super().find_package_modules(package, package_dir)
        return [(pkg, mod, file, ) for (pkg, mod, file, ) in modules if mod == '__init__']


setup(
    name="My_Pkg",
    version="0.0",
    ext_modules=cythonize('lib/my_pkg/my_mod.py'),
    packages=["my_pkg"],
    package_dir={'': 'lib'},
    cmdclass={'build_py': build_py},
)
Adriano
  • 487
  • 4
  • 8