2

I have an .so file called tissue-classifier.cpython-37m-x86_64-linux-gnu.so from an external library that I want to import so that I can extend it in one of my local classes. Since I am extending a class, I need to import it as an extension type using cimport and I am wondering if this is even possible. If I use a normal import statement then I will be left with a Python compiled version which cannot be used to extend a cdef class in my current file.

When I try to cimport the tissue_classifier file, it gives me the error that tissue_classifier.pxd file could not be found which makes sense since it is in .so format. Sorry if this is a dumb question, I just haven't been able to figure this out for a while.

Connor Johnson
  • 68
  • 5
  • You can import a .so file (if it's a suitably prepared for Python). However cimport is different: it happens when the module is compiled by Cython rather than when it is run and requires a .pxd file. You've been told (at least twice) that the problem is most likely with the Python path. I suggest you edit this question to show where the relevant files are stored and what you have tried with the path in setup.py. [This bit of documentation](https://cython.readthedocs.io/en/latest/src/userguide/sharing_declarations.html#search-paths-for-definition-files) might be useful. – DavidW Jun 26 '19 at 16:31
  • Sorry for the lack of clarification. I have solved the issue with the path when I found out that the package I wanted to import was not in `.pxd` format, but rather `.so` format. I am not getting that error anymore. I cannot import the `.pxd` file without getting it directly from Github which I don't know how to do. Instead, I am wondering if it is possible to import an extension type to extend a cdef class from this `.so` file. – Connor Johnson Jun 26 '19 at 17:06
  • 1
    No. Without the .pxd file you can't use cimport. You could inherit from the class normally, but not with a `cdef class`. [In your earlier question](https://stackoverflow.com/questions/56739276/how-to-import-external-pyx-libraries-into-my-pyx-file) you said "dipy/tracking/local contains both tissue_classifier.pyx and tissue_classifier.pxd" - what's changed? – DavidW Jun 26 '19 at 17:29
  • In the [Github repository](https://github.com/nipy/dipy/tree/master/dipy/tracking/local) in the specified directory, it has both the `.pxd` and `.pyx` file. However, when I decided to check the actual directory that I had installed, it showed me [this](https://imgur.com/a/vbXfYWN), where the files were all in `.so` format. Is there a way to import them directly from Github? – Connor Johnson Jun 26 '19 at 17:58
  • That makes a lot more sense now! If you only need one file the easiest thing might just be to copy it from github (use the "raw" button to get the original file and then save to the directory with the .so file); if that doesn't work then you might need to try non-anaconda ways of installing it (either pip or from source) – DavidW Jun 26 '19 at 18:56

1 Answers1

1

No, a *.so-file cannot be cimported.

If one is having C/CPP-backgrounds, then pyx/pxd/so-business is propably easiest to understand using the following model:

  • the resulting extension (*.so-file) corresponds to the final artefact in C/CPP-world which could be an executable, a shared-object (*.so), or a library/object-file collection. If you just runs the resulting program it is all you need. For example you can use (and probably do) a CPython-interpreter without having built it or having its source code. In analogy, if you have a binary extension (*.so) you can import and use it whitout having to build it (or even having corresponding pyx-files or a compiler on your machine) - that is what is provided by a wheel.
  • *.pyx corresponds to c/cpp-files, which have the definitions of the functionality. These files are needed if you want to build the resulting artifact from the source. In C/CPP-world this build process would be triggered by using make or similar. pyx-files are needed if you install the package via python setup.py install - which corresponds to calling make.
  • *.pxd corresponds to headers (h/hpp-files): it decribes the functionality in the resulting so-files, so it can be reused. For example just having CPython-interpreter isn't enough to build extensions - one has to install the dev-version so also the includes Python.h&Co. are present at the machine.

So what can be done?

First possibility:

If authors of the package consider *.pxd-files being part of the public API, then they can put the corresponding pxd-files next to *.so-files into the installation, so the c-interface of the module can be used/extended.

If they don't put the pxd-file into the installation, so chances are high that this c-interface is an implementation detail and you should not be using it, as it may change without notice in the future.

However, it is possible to take the risk and to copy the necessary pxd-files to the installation per hand, but making first sure that it is the right pxd-version (i.e. the same with which so-files in the installation were built).

Second possibility:

The easiest way to ensure, that the right pxd-version is used is to build package from the source, i.e.

  • dowloading the the right versioin from github (master or last release)
  • calling python setup.py install or what README tells you to do

Now, instead of copying pdx-files to the installation, one could add include_path to the downloaded package-source via include_path for cythonize-function or by adding it to sys.path.

Alternatively, as @BeforeFlight has pointed out in the comments, one could use python setup.py develop (or pip install -e the same folder so it can be deinstalled), and because it creates a link instead of copying data, the pxd-files will be found.

The above solutions will help to build the module, distributing it is a completely different story however.

ead
  • 32,758
  • 6
  • 90
  • 153
  • 1
    One may use `python setup.py develop` instead. So installations will be linked with cloned github repo and all `.pxd` files will be accessible. – Xronx Jul 10 '19 at 11:11