4

I try to implement the answer https://stackoverflow.com/a/57480599/7482208, but I am stuck on cimporting one package from another.

The code is here: https://github.com/iamishalkin/setuptools_cython_question

What I want is to have one independent package wrap from wrapper folder such that you can use it without cust package.

And I also want to be able to create custom functions by inheriting FuncWrapper class from wrap.

What I do:

  • Firstly I run python setup.py bdist_wheel in wrapper folder (this is what I am doing wrong, I suppose, as it gives me only binary file)
  • Next pip instal dist/(some_name).whl
  • Next I add import wrap and include_dirs=wrap.get_include() to custom/setup.py like it is done in numpy
  • I run python setup.py bdist_wheel in custom folder and this fails, on the first step no files except binary one were created

So the question is: how to add .pxd files to the final package.

I also tried sdist which does not compile cython code but just copies it.

0 _
  • 10,524
  • 11
  • 77
  • 109
Ivan Mishalkin
  • 1,049
  • 9
  • 25
  • Have you checked [my answer](https://stackoverflow.com/a/57483789/4720025) there? – Arda Aytekin Aug 14 '19 at 11:51
  • I think the answer starts from ["If you want to expose the C-level interface of your library for other libraries to cimport from, use package_data to install the .pxd files"](https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#distributing-cython-modules) however I've never _really_ understood setup.py and it just seems to ignore package data – DavidW Aug 14 '19 at 11:52
  • 1
    @ArdaAytekin It's a good approach to the specific problem (if you're allowed to change the C interface); the `BinaryOp` version would still need some way of installing .pxd files to be useful as a library (otherwise users can't easily inherit a `cdef class` from `BinaryOp` so it doesn't really avoid _this_ problem – DavidW Aug 14 '19 at 11:59
  • @ArdaAytekin Yes, I saw it, but I am not able to change c code unfortunately – Ivan Mishalkin Aug 14 '19 at 12:39
  • Relevant: https://stackoverflow.com/q/33555927/1959808 – 0 _ Jun 30 '21 at 03:17
  • Relevant: https://stackoverflow.com/q/56115159/1959808 – 0 _ Jun 30 '21 at 03:21

2 Answers2

5

As I said in a comment, the Cython documentation recommends putting .pxd files in package_data to install them. This necessitates a slightly different structure:

| setup.py
+ wrapper
   | wrap.pxd
   | wrap.pyx
   | __init__.py # just so it's recognised as a package
                 # may be unnecessary with recent Python versions

setup.py then creates a "package" called wrapper (this is modified from your version so it's possible it could be simplied further):

from setuptools import setup, Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext

NAME = "some_name"

ext_abc = Extension(name="wrapper.wrap",
                    sources=["wrapper/wrap.pyx"]
                    )

EXTENSIONS = [
    ext_abc
]

if __name__ == "__main__":
    setup(
        zip_safe=False,
        name=NAME,
        packages=["wrapper"],
        cmdclass={"build_ext": build_ext},
        ext_modules=cythonize(EXTENSIONS, language_level=3),
        package_data = {
            "wrapper": ["*.pxd"],
    },
        )

Note that I've changed the name of the extension to "wrapper.wrap" to ensure that it's installed as part of the package. The package_data is then able to recognised .pxd files as part of wrapper that you want to install. This doesn't work unless you put it in a "package".

You then install it. I just installed it with python3 setup.py install but I'm sure going through a wheel does largely the same thing.

For another module to use you file it's very simple:

from wrapper.wrap cimport FuncWrapper

The setup.py for that other module need have nothing special - you definitely don't need anything like include_dirs=wrap.get_include().

If you want to have an interface where you don't need submodules so can just do

from wrapper cimport FuncWrapper

then just use an __init__.py containing:

from .wrap import *

and an __init__.pxd containing:

from wrapper.wrap cimport * # relative import is a little broken in Cython I think

I'm sure there are other ways of doing this - I've only really used setuptools for compiling Cython stuff and never really worried about distributing too much so am not an expert - but this looks to be the standard approach.

DavidW
  • 29,336
  • 6
  • 55
  • 86
  • Thank you, I've also found example on github, but have not implemented this yet for my problem to answer https://github.com/jkleckner/cython_example – Ivan Mishalkin Aug 14 '19 at 14:37
  • Something strange is happening, when I add `from wrap cimport *` to `__init__.pxd` building custom I get `__init__.pxd:1:0: 'wrap.pxd' not found` and if I do not do this, then when I call `PyClass().py_wrap(add, 5, 5)` I get `Argument 'func' has incorrect type (expected wrap.FuncWrapper, got wrapper.wrap.FuncWrapper)` – Ivan Mishalkin Aug 15 '19 at 10:54
  • 1
    I've edited the answer so `__init__.p*` works (I also had to remove the slightly odd attempt to import itself from wrap.pxd) – DavidW Aug 15 '19 at 12:13
  • I have applied this changes but this does not solved the problem with types. I updated github repo with question. My collegue have solved this problem with poetry, and when I understand the solution I'll share it here – Ivan Mishalkin Aug 16 '19 at 10:11
0

In case anyone else runs into this problem, I was able to get this working using a MANIFEST.in file placed in the top-level directory. The contents are simply:

include MANIFEST.in
global-include *.pyx
global-include *.pxd
global-exclude *.pyc *.pyo *.pyd

I use this in addition to what the selected answer recommends.

user2520726
  • 1