Creating self contained wheel

Question

I try to create a self contained wheel for pyRFC: http://sap.github.io/PyRFC/install.html

I have the needed (closed sourced) libraries and header files.

Installing pyRFC works, if I copy the libraries and headers to $VIRTUAL_ENV/lib and $VIRTUAL_ENV/include.

Now I try to create a wheel which contains the closed source library files.

But I am failing.

If I unzip the wheel it only contains this:

(pypi)pypi@pypiserver:~> unzip -l packages/pyrfc-1.9.91-cp27-cp27mu-linux_x86_64.whl
Archive:  packages/pyrfc-1.9.91-cp27-cp27mu-linux_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
     5366  2018-07-23 13:38   pyrfc/_exception.py
     1045  2018-07-23 13:38   pyrfc/__init__.py
  1610216  2018-07-23 14:10   pyrfc/_pyrfc.so
     3835  2018-07-23 14:10   pyrfc-1.9.91.dist-info/DESCRIPTION.rst
      990  2018-07-23 14:10   pyrfc-1.9.91.dist-info/metadata.json
        6  2018-07-23 14:10   pyrfc-1.9.91.dist-info/top_level.txt
      105  2018-07-23 14:10   pyrfc-1.9.91.dist-info/WHEEL
     4666  2018-07-23 14:10   pyrfc-1.9.91.dist-info/METADATA
      715  2018-07-23 14:10   pyrfc-1.9.91.dist-info/RECORD
---------                     -------
  1626944                     9 files

The extra_objects (see below) are missing.

How can I modify the setup.py of pyRFC to make the wheel contain the libraries from $VIRTUAL_ENV/lib?

Here is the setup.py: https://github.com/SAP/PyRFC/blob/master/setup.py

I tried this patch for setup.py

@@ -48,7 +49,9 @@ PYRFC_EXT = Extension(
     , libraries=LIBS
     , define_macros=MACROS
     , extra_compile_args=COMPILE_ARGS
-    , extra_link_args=LINK_ARGS
+    , extra_link_args=LINK_ARGS,
+      library_dirs=['lib'],
+      extra_objects = ['lib/libicudata.so.50', 'lib/libsapnwrfc.so', 'lib/libicui18n.so.50', 'lib/libicuuc.so.50', 'lib/libicudecnumber.so', 'lib/libsapucum.so'],
 )

If I install and run the library without libsapnwrfc.so I get this error:

Traceback (most recent call last):
  File "test-pyrfc.py", line 1, in <module>
    from pyrfc import Connection
  File "/home/other/lib/python2.7/site-packages/pyrfc/__init__.py", line 22, in <module>
    from pyrfc._pyrfc import get_nwrfclib_version, Connection, TypeDescription, FunctionDescription, Server
ImportError: libsapnwrfc.so: cannot open shared object file: No such file or directory

If I trace the open calls, I see that it only looks at root-level for the library. The library does not get search in the virtualenv (which is /home/other):

strace python test-pyrfc.py 2>&1 | grep libsapnwrfc.so
open("/lib64/tls/x86_64/libsapnwrfc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/tls/libsapnwrfc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/x86_64/libsapnwrfc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/libsapnwrfc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib64/tls/x86_64/libsapnwrfc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib64/tls/libsapnwrfc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib64/x86_64/libsapnwrfc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib64/libsapnwrfc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
write(2, "libsapnwrfc.so: cannot open shar"..., 73libsapnwrfc.so: cannot open shared object file: No such file or directory) = 73

I would like to install "libsapnwrfc.so" via wheel into a virtualenv because this gives me the power to have several independent environments. I could install it via RPM or config-management, but I would like to avoid it.

@AzatIbrakov you mean absolute paths for extra_objects? I read the docs. This kwarg gets used for linking. That's not the problem. I was on the wrong track here. I will look at this again tomorrow. — guettli, Jul 23 '18 at 14:45
I'd suggest to use `audtiwheel` for including all the necessary libs in the wheel. Check out the question [How to build and distribute a Python/Cython package that depends on third party libFoo.so](https://stackoverflow.com/questions/47042483/how-to-build-and-distribute-a-python-cython-package-that-depends-on-third-party); it has an excellent answer on repairing the wheel with `auditwheel`. — hoefling, Jul 24 '18 at 17:22
@hoefling auditwheel looks good. If you write your comment as answer, I will upvote it. — guettli, Jul 25 '18 at 08:00

score 3 · Accepted Answer · answered Jul 25 '18 at 17:25

The extra_objects are used only when linking the extension lib and are not bundled in the wheel. From the docs:

extra_objects: list of extra files to link with (eg. object files not implied by ‘sources’, static library that must be explicitly specified, binary resource files, etc.)

For bundling the libs into wheel, use auditwheel for Linux wheels, or delocate for MacOS wheels. SO already has an excellent question: How to build and distribute a Python/Cython package that depends on third party libFoo.so which covers the whole wheel repairing process using both auditwheel and delocate. Here, I will only summarize the necessary commands. The process is pretty similar with both tools:

$ pip install auditwheel  # you may also need to install patchelf
$ python setup.py bdist_wheel
$ auditwheel show dist/*_linux_x86_64.whl  # will show the libs to be bundled
$ auditwheel repair dist/*_linux_x86_64.whl

This will create a new dir dist/wheelhouse, containing the new wheel with the bundled libs.

Creating self contained wheel

1 Answers1