2

I have a shared library that I have created using pybind11.

called dddd.so

when I open python and type

import dddd
print(dddd.test_function())

I get my desired result, and that's fantastic.

However, my goal is to create a conda installable package, so that I could create a conda environment, and then type:

conda install dddd

And have that work.

So I am trying to create a setup.py file that will allow me to create this using python setup.py bdist or python setup.py bdist_conda

I have not been able to get my shared library to go into a bdist module.

My directory structure at the moment is

dddd/
    - dddd/
           -dddd.so
           -__init__.py
setup.py

I have tried a lot of things, including Distribute a Python package with a compiled dynamic shared library , but I have not been able to get it to work.

When I try this as my setup.py:

from distutils.core import setup
setup(
    name='dddd',
    version='0.2',
    packages=['dddd'],
    package_dir={'dddd': 'dddd'},
    package_data={'dddd': ['dddd.so']},
)

I am able to get python setup.py bdist_conda to work.

However when I run this conda module, my module can now not be directly accessed, I have to do:

import dddd.dddd
dddd.dddd.test_function()

Which is not what I want.

I can't seem to achieve what was achieved in the answer in my link.

dddd4
  • 136
  • 7
  • Why is this tagged `C++` and `pybind11`? – Reblochon Masque Jul 31 '18 at 07:10
  • 1
    So it is a precompiled shared object. Make `dddd` to a package by adding an `__init__.py`, add `dddd` to `packages` list in `setup.py` and include `dddd.so` via `package_data` like described in the question you linked. `Extension`s are used when you have the source code for the shared library and compile it to shared object when building the distribution. – hoefling Jul 31 '18 at 11:41
  • @hoefling Thanks, have updated with package_data, but can't seem to directly access my module – dddd4 Jul 31 '18 at 21:49
  • This is exactly what is done in the linked question. If you want to place the shared object directly to site-packages, then your best bet is to build it from C sources by declaring an `Extension`. You can only place python modules or `Extension`s in the site-packages root; although there are some (hacky) ways to circumvent that, they all have exceptions and won't work in all situations. – hoefling Jul 31 '18 at 22:53
  • Alternatively, you can declare imports in the `__init__.py` file, something like `from .dddd import test_function`, so it is already imported at the `dddd` package level; this way, you should be able to do `import dddd; dddd.test_function()` in the calling modules. – hoefling Jul 31 '18 at 22:58
  • @hoefling the command `from .dddd import test_function` in init.py achieves the end result I am after, although I guess it means I have to write that for every function in my dddd.so file. At least it works. If you want to put that comment as an answer I will accept it – dddd4 Aug 01 '18 at 00:04
  • 1
    Sure, it's a major drawback as you need to keep your `__init__.py` consistent on changes; but why don't you want to build the shared object from C source code by configuring an `Extension`? This would be the most straightforward way IMO. Should you need a prebuilt shared object in `dddd` for development, you can always build one via `python setup.py build_ext`. You would then also eliminate the need of the `dddd` package completely and your imports would work by default. – hoefling Aug 07 '18 at 10:23
  • @hoefling yes an extension would probably be a better way to do this. I will probably head in this direction. My initial reluctance was that I already build my project using a build system (meson) and I didn't want to go down the path of having another file where I needed to specify include directories, source files etc.. but comparing that hassle to maintaining an init.py and it's not much different. I could either get my setup.py file to use pkg-config or use a [custom target](http://mesonbuild.com/Generating-sources.html) in meson to streamline my process – dddd4 Aug 07 '18 at 22:55
  • Interesting - last time I checked, `meson` couldn't start anything with Python projects (funny because it's written in Python itself). Now it looks like [some rudimentary support was implemented](http://mesonbuild.com/Python-module.html). I'm curious whether it's enough to build wheels with `meson`? Wheels don't need a setup script, all you need is to generate valid package metadata files (dist-info). Looks like one could write a custom target that generates the metadata, then zip the shared object along with the metadata, rename zip to wheel and done. – hoefling Aug 08 '18 at 15:38

0 Answers0