8

My project has the following directory structure:

.
├── Makefile
├── pxd
├── pyx
│   ├── Landscaping.pyx
│   ├── Shrubbing.pxd
│   └── Shrubbing.pyx
└── setup.py

However, if I move Shrubbing.pxd anywhere else, say, into pxd/, I get the following error:

Error compiling Cython file:
------------------------------------------------------------
...
import pyx.Shrubbing
cimport Shrubbing
       ^
------------------------------------------------------------

pyx/Landscaping.pyx:2:8: 'Shrubbing.pxd' not found

Error compiling Cython file:
------------------------------------------------------------
...
import pyx.Shrubbing
cimport Shrubbing

cdef Shrubbing.Shrubbery sh
    ^
------------------------------------------------------------

This is strange because in setup.py I have:

from distutils.core import setup, Extension
from Cython.Build import cythonize
setup(ext_modules=cythonize([
    Extension(
        'pyx.Landscaping',
        sources=["pyx/Landscaping.pyx"],
        include_dirs=['pxd']), # <-- HERE
    Extension('pyx.Shrubbing', sources=["pyx/Shrubbing.pyx"])
]))

which clearly specifies the new directory for Shrubbing.pxd.

The source files are all very short, but to avoid cluttering this post, I will just post a link to a repository: https://github.com/lobachevzky/landscaping

Thanks for your help.

ethanabrooks
  • 747
  • 8
  • 19

2 Answers2

5

include_dirs is for C/C++ headers, not Cython pxd files.

In general it is best to keep related pyx/pxd files together in same directory, ie Shrubbing.pyx and Shrubbing.pxd should be in same directory.

To then use that from other modules, include a __init__.pxd and cimport via the name used in the Extension, eg pyx.Shrubbing as you would with Python modules.

If importing in python (not cimport), __init__.py should be included as well.

When using in the same module, OTOH, the .pxd needs to be available at runtime of that module, which means including it in Python search path.

If you want to organise the pxd files into separate dirs then link them symbolically within the module directory to make them available to the module, the dir containing an __init__.pxd or .py file.

It's a bit messy as Cython does not currently support relative imports, hence the need for linking when wanting to import from another dir.

See documentation for more details.

danny
  • 5,140
  • 1
  • 19
  • 31
  • I think this is the answer I am looking for, but what do you mean by 'link them within the module directory'? Is that a `cimport` statement or do you actually mean with a symbolic link? – ethanabrooks Nov 25 '17 at 14:07
  • @ethanabrooks Symbolic link, yes. – danny Apr 18 '18 at 10:21
  • I don't think a symbolic link is necessary. In the [Cython docs](http://cython.readthedocs.io/en/latest/src/userguide/sharing_declarations.html#search-paths-for-definition-files), as long as you include the `__init__.pxd` in the directory where your `.pxd` is located, Cython will search your path for the definitions file when you use `cimport` in an implementation, `.pyx` file. I've used this to create some linear algebra wrappers, which I import in project specific cython implementation files. So just make sure the `.pxd` file is in a directory that's in your path, e.g., by setting PYTHONPATH – jtorca May 09 '18 at 04:10
  • Per the answer, symbolic links are only necessary `If you want to organise the pxd files into separate dirs`, meaning from where the module directory containing `__init__.pxd` resides. – danny May 09 '18 at 10:30
1

Strictly speaking this answer is broader than the specific question you are asking. It is more "how do you share pxd definitions between packages".

To sort this out for myself, I created a cython_example repo. It is not satisfying because it uses the old distutils tools rather than setuptools to prevent zip/egg from encapsulating the underlying files but may shed some light here. I'm actively looking for an improvement that will work with the automatic extraction of code from egg files so that setuptools can be used.

The core idea is to use the os.path.dirname(from_package.__file__) to use to add to the include path. See the setup.py file.

Update:

The cython email list provided the hint that using zip_unsafe when installing using setuptools would make everything work. See the updated working example and the update I added to the documentation .

Traveler
  • 1,048
  • 14
  • 27
  • So danny, the other answerer, indicated that `include_dirs` is for header files, not `pxd ` files. Do you disagree with that assertion? – ethanabrooks Apr 05 '18 at 00:44
  • If you don't zip the files, the ordinary include mechanisms of (python,cython) work and don't require `include_dirs` to be used. So in a way, he is correct in that you need to manually add dirs to the path that won't be found by python dependencies that are rooted in a non-zip directory. I will update my repo to reflect that. – Traveler Apr 05 '18 at 05:20
  • From the [actual documentation](http://cython.readthedocs.io/en/latest/src/reference/compilation.html?highlight=include_dirs#configuring-the-c-build): `The Extension class takes many options, and a fuller explanation can be found in the distutils documentation. Some useful options to know about are include_dirs, libraries, and library_dirs which specify where to find the .h and library files when linking to external libraries.` – danny Apr 25 '18 at 10:35
  • From same documentation: `Note also that if you use setuptools instead of distutils, the default action when running python setup.py install is to create a zipped egg file which will not work with cimport for pxd files when you try to use them from a dependent package. To prevent this, include zip_safe=False in the arguments to setup().` – danny Apr 25 '18 at 10:36
  • `include_dirs` has nothing to do with either `cimport` or `.pxd` files. See documentation as linked above. – danny Apr 25 '18 at 10:38
  • 1
    @danny Thank you for linking the documentation. I was the one who created that patch for the documentation. ;-) – Traveler Apr 27 '18 at 16:07