3

I have three questions, but related, and I not getting how can I split them well. I've found many information about those issues, like submodule extension, hierarchy, about an empty __init__.py file, and how to cythonize multiple pyx files. But when I try them together I cannot make them work.

I've prepared an small repo thinking in place together code samples of issues solved. I've check even the code of some of the projects listed that uses cython, but still not getting how to does three things at the same time.

  • Empty __init__.py file:

In a project (where all the files are pyx (and pxd ifneedbe)), with a __init__.pyx that includes all of them, when there is a __init__.py file, then import it doesn't load the ".so" but the empty init.

  • cythonize multiple files:

When instead of prepare a __init__.py that includes all the elements of a module. Like:

cythonize(['factory.pyx', 'version.pyx'])

the resulting ".so" import raises an exception:

>>> import barfoo
(...)
ImportError: dynamic module does not define init function (PyInit_barfoo)

It would be related with the previous question if it is necessary to write something in __init__.py.

  • Submodule:

In fact, this is the main question. The singleton.pyx would part of a submodule, lets say utils to be used from other elements in the module.

For the sample there is a submodule (simply called subm) added in the setup.py as another extension. I've placed earlier than the main one (I don't know if this really does any difference, I didn't see it).

>>> import barfoo
>>> import barfoo.subm
(...)
ImportError: No module named 'barfoo.subm'

Separately, those recipes work, but together I cannot. The "__init__.py" seems to be necessary when there is a mix of "py" and "pyx" files. The examples explain how to cythonize with multiple files, but don't include the last key point for the import. And the submodules doesn't complete about how they can be imported from one place or another (import submodules when import the base one, or optional imports when they are specified).

Community
  • 1
  • 1
srgblnch
  • 323
  • 3
  • 11
  • 1
    Can you describe your file layout (before and after compilation)? I'm not 100% clear from the question but I think you're trying to compile multiple .pyx files to a single .so file, and that doesn't work. Each .pyx file needs to generate its own .so file – DavidW May 18 '17 at 10:30

2 Answers2

2

Last night I saw your question, and made a simple example according to the wiki. But that question was deleted quickly.

Here is the example: https://github.com/justou/cython_package_demo

Make sure the settings of C compiler and python environment is correct, compile the pyx files by run:

python setup.py build_ext --inplace

Usage is the same as python package:

from dvedit.filters import flip, inverse, reverse
flip.show()       # print: In flip call Core function
inverse.show()    # print: In inverse call Core function
reverse.show()    # print: In reverse call Core function

BTW, there is no need to create an __init__.pyx, you can do the ext_module importings in the __init__.py file

oz1
  • 938
  • 7
  • 18
  • After post it yesterday I thought to do another iteration by myself and try to write it better. Sorry and thanks. – srgblnch May 18 '17 at 10:45
  • Almost, I think I've got something. The inplace left the so file in the dvedit directory, but in my try left it one below (where the setup is). – srgblnch May 18 '17 at 14:39
  • @srgblnch I've been faced with the same problem in the past under windows. By updating setuptools, changing the directory, restarting pc, the problem gone away, I don't know wether it's a bug, and can't figure out why, even can't reproduce it now! I highly doubt it's a problem about 'path'. – oz1 May 18 '17 at 16:22
  • With the comment on the question, I think, we've got the key about the directory and the \_\_init\_\_.py. *import dvedit* refers the python, but *import dvedit.core* refers the so comming from the core.pyx. – srgblnch May 19 '17 at 07:30
  • As a followup, I am trying to do something similar, except, I want to create a submodule with cdef definitions and pxd files. Now I want to use this submodule in another module that uses these cdef definitions. Unfortunately, I fail at doing so, and seem to be unable to access them without them being cpdef'd. – axolotl Apr 08 '20 at 12:33
2

Thanks to the comments from oz1 and DavidW, I've got the solution. Yes, those three things come together.

  • Very important is the order when import in the setup.py. Even the PEP8 doesn't say that the imports should be alphabetically sorted, there are other guide lines (like reddit's) that does and I usually follow:

When import first cythonize and then setup, will cause that when cythonize(find_pyx()) is called, the result will be a list of distutils.extension.Extension objects.

from setuptools import setup, find_packages
from Cython.Build import cythonize

setuptools must be imported before cython and then the result of cythionize() will be a list of setuptools.extension.Extension objects that can be passed to the setup() call.

  • Important to understand the meanings of the __init__'s:

All the __init__.pyx files with includes has been removed and each .pxy file produces its own .so binary.

The main module and the submodules will exist as long as their directories contain the __init__.py file like happen with a pure python code.

In the example I've linked, the file barfoo/__init__.py is not empty because I want that import barfoo provides access to elements like version() or Factory(). Then, this __init__.py is who imports them like a normal pure python.

  • For the submodule:

Similar for the submodule and its own __init__.py file. In this example the import barfoo will do a from .factory import Factory, who will call from barfoo.subm import Bar, then the subm will be available.

But if the submodule is not imported in this secondary way, the user will have access to it with calls like import barfoo.subm.

srgblnch
  • 323
  • 3
  • 11