Compile and use python-openzwave with open-zwave in non-standard location

Question

I manually compiled python-openzwave to work with C++ library.

I would like to use it as Kodi addon (OpenELEC running on Pi 3), so can not use standard installation. I've compiled everything, downloaded missing six and louie libs, and now try to run hello_world.py.

My current dirs structure is the following:

- root
  - bin
      - .lib
      - config
        Alarm.o
        ...
        libopenzwave.a
        libopenzwave.so
        libopenzwave.so.1.4
        ...
  - libopenzwave
      driver.pxd
      group.pxd
      ...
  - louie
      __init__.py
      dispatcher.py
      ...
  - openzwave
      __init__.py
      command.py
      ...
  six.py
  hello_world.py

But when I run hello_world.py, I get the following error -

Traceback (most recent call last):
  File "hello_world.py", line 40, in <module> 
    from openzwave.controller import ZWaveController 
  File "/storage/.kodi/addons/service.multimedia.open-zwave/openzwave/controller.py", line 34, in <module> 
    from libopenzwave import PyStatDriver, PyControllerState 
ImportError: No module named libopenzwave

If I move libopenzwave.a and libopenzwave.so to root folder, then I get the following error:

Traceback (most recent call last):
  File "hello_world.py", line 40, in <module> 
    from openzwave.controller import ZWaveController 
  File "/storage/.kodi/addons/service.multimedia.open-zwave/openzwave/controller.py", line 34, in <module> 
    from libopenzwave import PyStatDriver, PyControllerState 
ImportError: dynamic module does not define init function (initlibopenzwave)

What is wrong with my setup?

Are you sure that your module has initialization info, as said here: https://docs.python.org/3/extending/extending.html#the-module-s-method-table-and-initialization-function ? — noteness, Jun 13 '16 at 03:52
Manually compiled how exactly? Not finding `initlibopenzwave` is caused, if I am not mistaken, by the `Cython` files not being `cythonized` and compiled correctly (`Cython` takes care of creating the `init` functions for you. Could you elaborate on the compilation process? — Dimitris Fasarakis Hilliard, Jun 15 '16 at 06:21
@Jim, I've downloaded the source https://github.com/OpenZWave/python-openzwave/raw/master/archives/python-openzwave-0.3.1.tgz, unpacked it and run `make build`. This version doesn't require `Cython` in accordance with this discussion - https://groups.google.com/forum/#!topic/python-openzwave-discuss/Yrgf-Xg5XRk. — LA_, Jun 15 '16 at 17:02
@ShamilKMuhammed, that is not my module, but as I can see `libopenzwave.cpp` has `initlibopenzwave` function. — LA_, Jun 15 '16 at 17:41
The only case where I see the `.pyx` files getting built is during `setup-lib.py` which is run when `sudo make install` (or when `sudo make install-lib`) is executed. I just get a lovely segfault on `import` now. — Dimitris Fasarakis Hilliard, Jun 15 '16 at 18:15
`libopenzwave` has been generated so indeed there does not seem to be a need to use `Cython` (for `cythonizing` the file). You do need to build the generated file and I don't think `make build` does that. It just builds `openzwave` and not the `Python` extention for it. — Dimitris Fasarakis Hilliard, Jun 15 '16 at 18:21
@Jim, thanks (never worked with C++ libs from Python before). What is the command to build `.pyx` files? Looking at `setup-lib.py` I can not find it. — LA_, Jun 15 '16 at 18:25
`sudo make install` will call `setup-lib.py` for you automatically. It is going to use the default `Python` install directory for installations. — Dimitris Fasarakis Hilliard, Jun 15 '16 at 18:27
@Jim, this is the problem - since I am going to use that as OpenELEC/Kodi addon, I can not use standard `make install`. I should cross compile everything on Ubuntu, then use the archive on OpenELEC without any `install` steps. — LA_, Jun 15 '16 at 18:33
@LA_ ah, right. The only thing I can think of is compiling `libopenzwave.cpp` with `python setup-lib.py build_ext --inplace`. This will compile it *inplace* (creating the shared library in the `src-lib` folder) You could then try adding it in the folder you need to import it from. (I have no familiarity with `OpenELEC` fwi, I just have experience with `cython`). — Dimitris Fasarakis Hilliard, Jun 15 '16 at 18:42
@Jim, I just found another way to generate the .c file - http://docs.cython.org/src/reference/compilation.html. The question now is how to cross compile it. — LA_, Jun 15 '16 at 18:51
`Cython` will just compile `Python` code (or `Cython` code) to a `.c` or `.cpp` file for you. From there on you should compile it with a standard `C/C++` compiler; this is what `setup-lib.py` does for you (it compiles it to a shared object) with `python setup-lib.py build_ext --inplace`. I see `OpenELEC` is a `Linux` system so compiling on a system like `Fedora`, `Ubuntu`, and others might just be what's needed. Have you tried running the `setup-lib.py` script? — Dimitris Fasarakis Hilliard, Jun 15 '16 at 18:52
@Jim, if I try to run `python setup-lib.py build_ext --inplace` on Ubuntu (where I cross compile), then I get the following error - `openzwave/libopenzwave.a: error adding symbols: File in wrong format`. — LA_, Jun 15 '16 at 19:32
@Jim, looks like I don't need to do anything with `Cython`.. `libopenwave.cpp` file which comes in the package I downloaded already has a comment that it is generated by Cython. — LA_, Jun 15 '16 at 19:45
Just learned that there is one more file created - `libopenzwave.o` in the `build` folder. Not sure yet how to use it. — LA_, Jun 15 '16 at 21:57
Yes, `Cython` is definitely not needed. The error with `setup-lib.py` is probably because `setup-lib.py` isn't being compiled in the same way as the rest of the files in `openzwave`. When running `make build` do you set some specific flags in order to get cross compilation? — Dimitris Fasarakis Hilliard, Jun 16 '16 at 04:27
@Jim, yes, I pass ARCH and CROSS_COMPILE values. Interesting moment - I tried to repeat my steps of manual installation on Ubuntu (without cross compilation) and when run the test python code, got 'segmentation fault' error. Strange, that I don't get the same error about missing init function. — LA_, Jun 16 '16 at 08:41
@LA_ yes, same segfault here, don't know what causes it yet, it finds the `init` function in this case because you compile everything using the same flags. [You may override/provide flags to `setup-lib.py`](http://stackoverflow.com/questions/6928110/how-may-i-override-the-compiler-gcc-flags-that-setup-py-uses-by-default) as another attempt. — Dimitris Fasarakis Hilliard, Jun 16 '16 at 09:04
@Jim, I've stopped my experiments and decided to work with open-zwave with C++ directly instead of Python. Could you please post something as the answer, so I will be able to accept it and give you a bounty ;). — LA_, Jun 18 '16 at 08:03
@LA_ I added an answer. Either case, if you ever bump into this problem with `python-openzwave` again and have an interest in a more complete answer, I would gladly place the same bounty on this question to respark interest and probably get a new answer. — Dimitris Fasarakis Hilliard, Jun 18 '16 at 14:14

score 2 · Accepted Answer · edited May 23 '17 at 12:33

In general the steps required consist of calls to make build which handles building the .cpp files for openzwave and downloading all dependencies (including Cython); and make install which runs the setup-api, setup-lib.py (this setup script also creates the C++ Python extention for openzwave), setup-web.py and setup-manager.py.

Since you cannot run make install as you specified and are instead using the archive they provide, the only other options for creating the python extention, after building the openzwave library with make build, is generating the .so files for it without installing to standard locations.

Building the .so for the cython extention in the same folder as the Cython scripts is done by running:

python setup.py build_ext --inplace

This should create a shared library in src-lib named libopenzwave.so (it is different from the libopenzwave.so contained in the bin/ directory) which contains all the functionality specified in the extention module. You could try adding that to the libopenzwave folder.

If you pass special compiler flags during make build for building the openzwave library you should specify the same ones when executing the setup-lib.py script. This can be done by specifying the CFLAGS before executing it (as specified here) or else you might have issues like error adding symbols: File in wrong format.

score 1 · Answer 2 · edited May 23 '17 at 11:45

Here's the description of the python-openzwave's build from the question's perspective. Almost all the steps correspond to the root Makefile's targets.

Prerequisites. There are several independent targets with little to no organization. Most use Debian-specific commands.
- Cython is not needed if building from an archive (details below)
openzwave C++ library (openzwave openzwave/.lib/ target).
- Build logic: openzwave/Makefile, invoked without parameters (but with inherited environment).
- Inputs: openzwave/ subtree (includes libhidapi and libtinyxml, statically linked).
- Outputs: openzwave/.lib/libopenzwave.{a,so}
- Accepts PREFIX as envvar (/usr/local by default)
  - The only effect that affects us is: $(PREFIX)/etc/openzwave/ is assigned to a macro which adds a search location for config files (Options.cpp): config/ -> /etc/openzwave/ -> <custom location>.
libopenzwave Python C extension module (install-lib target - yes, the stock Makefile cannot just build it; the target doesn't even have the dependency on the library).
- Build logic: setup-lib.py
- Inputs: src-lib/, openzwave/.lib/libopenzwave.a
- Outputs: build/<...>/libopenzwave.so - yes, the same name as openzwave's output, so avoid confusing them
  - By default, openzwave is linked statically with the module so you don't need to include the former into a deployment
  - The module does, however, need the config folder from the library. It is included by the build script when making a package.
- Contrary to what Jim says, Cython is not needed to build from an archive, the archive already includes the generated .cpp.
- Now, the catch is: the module itself uses pkg_resources to locate its data. So you cannot just drop the .so and config into the currect directory and call it a day. You need to make pkg_resources.get_distribution('libopenzwave') succeed.
  - pkg_resources claims to support "normal filesystem packages, .egg files, and unpacked .egg files."
  - In particular, I was able to pull this off: make an .egg (setup-lib.py bdist_egg), unpack it into the current directory and rename EGG-INFO into libopenzwave.egg-info (like it is in site-packages). A UserWarning is issued if I don't specifically add the .so's location into PYTHON_PATH/sys.path before importing the module.
openzwave,pyozwman and pyozwweb Python packages (install)
- these are pure Python packages. The first one uses the C extension module, others use the first one.
- Build logic: setup-api.py,setup-manager.py,setup-web.py
- Input: src-*/
- Output: (pure Python)
- They only use pkg_resources.declare_namespace() so you're gonna be fine with just the proper files/dirs on sys.path without any .egg-info's

Compile and use python-openzwave with open-zwave in non-standard location

2 Answers2