3

I'm building a Rust binary (liblonlat_bng.dylib) on Travis CI, pulling it into a Cython extension (in the same dir as the Cython source .c/.pyx), and testing it, also on Travis CI (in a different repo and build). However, tests of the Python package are failing, and I'm not sure why:

----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/Users/travis/build/urschrei/convertbng/test/test_convertbng.py", line 15, in <module>
    from convertbng.cutil import convert_bng as cconvert_bng
ImportError: dlopen(/Users/travis/build/urschrei/convertbng/convertbng/cutil.so, 2): Library not loaded: /Users/travis/build/urschrei/lonlat_bng/target/x86_64-apple-darwin/release/liblonlat_bng.dylib
  Referenced from: /Users/travis/build/urschrei/convertbng/convertbng/cutil.so
  Reason: image not found

Here's the OSX build output for the package from Travis:

Installing collected packages: convertbng
  Running setup.py develop for convertbng
    Running command /Users/travis/build/urschrei/convertbng/venv/bin/python2.7 -c "import setuptools, tokenize;__file__='/Users/travis/build/urschrei/convertbng/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop --no-deps
    running develop
    running egg_info
    writing requirements to convertbng.egg-info/requires.txt
    writing convertbng.egg-info/PKG-INFO
    writing top-level names to convertbng.egg-info/top_level.txt
    writing dependency_links to convertbng.egg-info/dependency_links.txt
    warning: manifest_maker: standard file '-c' not found
    reading manifest file 'convertbng.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'convertbng.egg-info/SOURCES.txt'
    running build_ext
    building 'convertbng.cutil' extension
    creating build
    creating build/temp.macosx-10.11-x86_64-2.7
    creating build/temp.macosx-10.11-x86_64-2.7/convertbng
    clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -Iconvertbng -I/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c convertbng/cutil.c -o build/temp.macosx-10.11-x86_64-2.7/convertbng/cutil.o -O3
    [unused function warnings]
    creating build/lib.macosx-10.11-x86_64-2.7
    creating build/lib.macosx-10.11-x86_64-2.7/convertbng
    clang -bundle -undefined dynamic_lookup build/temp.macosx-10.11-x86_64-2.7/convertbng/cutil.o -L. -Lconvertbng -llonlat_bng -o build/lib.macosx-10.11-x86_64-2.7/convertbng/cutil.so
    copying build/lib.macosx-10.11-x86_64-2.7/convertbng/cutil.so -> convertbng
    Creating /Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/convertbng.egg-link (link to .)
    Adding convertbng 0.4.14 to easy-install.pth file
    Installed /Users/travis/build/urschrei/convertbng
Successfully installed convertbng

And here's the Linux Travis output, which successfully locates the dylib on a relative path. Note the $ORIGIN argument to -R, which can't be used on OSX:

Installing collected packages: convertbng
  Running setup.py develop for convertbng
    Running command /usr/bin/python -c "import setuptools, tokenize;__file__='/home/travis/build/urschrei/convertbng/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop --no-deps
    running develop
    running egg_info
    writing requirements to convertbng.egg-info/requires.txt
    writing convertbng.egg-info/PKG-INFO
    writing top-level names to convertbng.egg-info/top_level.txt
    writing dependency_links to convertbng.egg-info/dependency_links.txt
    warning: manifest_maker: standard file '-c' not found
    reading manifest file 'convertbng.egg-info/SOURCES.txt'
    writing manifest file 'convertbng.egg-info/SOURCES.txt'
    running build_ext
    building 'convertbng.cutil' extension
    creating build
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/convertbng
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I. -Iconvertbng -I/usr/include/python2.7 -c convertbng/cutil.c -o build/temp.linux-x86_64-2.7/convertbng/cutil.o -O3
    creating build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/convertbng
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/convertbng/cutil.o -L. -Lconvertbng -Wl,-R$ORIGIN -llonlat_bng -o build/lib.linux-x86_64-2.7/convertbng/cutil.so
    copying build/lib.linux-x86_64-2.7/convertbng/cutil.so -> convertbng
    Creating /usr/local/lib/python2.7/dist-packages/convertbng.egg-link (link to .)
    Adding convertbng 0.4.14 to easy-install.pth file
    Installed /home/travis/build/urschrei/convertbng
Successfully installed convertbng

Other details:

  • Both my OS X and Travis are using XCode 7.3
  • The binaries are being built with the same commit both locally and on Travis
  • If I build the binary locally, then run setup.py build_ext --inplace on my package, the tests pass
  • I run the same install for Linux, using a .so built with the same commit on Travis, and the tests pass
  • If I copy the Travis-built .dylib to my local machine, then run setup.py build_ext --inplace, the tests fail with the same error as on Travis.

I'm very puzzled as to what's going on. Is there something I should be looking at in the binaries? The otool -l output differs slightly. For instance:

Local:

cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Thu Jan  1 01:00:02 1970
current version 1226.10.1

Travis:

cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Thu Jan  1 00:00:02 1970
current version 1225.1.1

Link to passing Linux job: https://travis-ci.org/urschrei/convertbng/jobs/136730347
Link to failing OSX job: https://travis-ci.org/urschrei/convertbng/jobs/136730348

UPDATE: It's definitely because of the library name. if I use install_name_tool to change the dylib location to @loader_path/liblonlat_bng.dylib:

install_name_tool -change /Users/travis/build/urschrei/lonlat_bng/target/x86_64-apple-darwin/release/liblonlat_bng.dylib @loader_path/liblonlat_bng.dylib convertbng/cutil.so

The Travis-built executable will run on my local machine. However, Travis seems to have a broken install_name_tool install, and I'm not managing to pass the correct invocation to cutil.so from setup.py. I've tried setting extra_link_args to ['-Wl,-rpath,'+'@loader_path/liblonlat_bng.dylib'], which generates

/usr/bin/clang -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -g build/temp.macosx-10.6-intel-2.7/convertbng/cutil.o -L. -Lconvertbng -llonlat_bng -o build/lib.macosx-10.6-intel-2.7/convertbng/cutil.so -Wl,-rpath,@loader_path/liblonlat_bng.dylib

But my tests still fail if I do that.

UPDATE 2: I can fix the install_name of the dylib in the link phase when building with cargo:

RUSTFLAGS="-C link-args=-Wl,-install_name,@rpath/liblonlat_bng.dylib" cargo build --release

And this works. But I suspect my setup.py Extension setup is wrong:

# only append the runtime dir on Linux
rdirs = []
ldirs = []
if sys.platform != 'darwin':
    # from http://stackoverflow.com/a/10252190/416626
    # the $ORIGIN trick is not perfect, though
    rdirs = ['$ORIGIN']
if sys.platform == 'darwin':
    ldirs = ['-Wl,-rpath,'+'@loader_path/liblonlat_bng.dylib']

extensions = Extension("convertbng.cutil",
                    sources=["convertbng/cutil" + suffix],
                    libraries=["lonlat_bng"],
                    include_dirs=['.', 'convertbng'],
                    library_dirs=['.', 'convertbng'],
                    runtime_library_dirs=rdirs,
                    extra_compile_args=["-O3"],
                    extra_link_args=ldirs
)
urschrei
  • 25,123
  • 12
  • 43
  • 84
  • Your question is unclear to me — I can't tell *which* package you are creating, and whether it has dependencies or anything like that. A [mcve] would be extremely helpful. Otherwise, the best advice I can give is "yeah, remove the absolute path or ensure that the library is installed at the exact same absolute path". – Shepmaster Jun 10 '16 at 20:20
  • @shepmaster I'm creating a Python package called convertbng. Its only dependency is on the dylib, which resides in the same dir as the generated cython .so – the problem is that even though I think I'm passing the correct arguments to my Extensions instance in setup.py, it's building with an absolute path. I've set the install_name to a relative path when building with cargo, but it's a workaround. – urschrei Jun 10 '16 at 20:25
  • But you aren't showing us (or it's very hard to see) how the dylib is installed in both builds, how the dylib is linked to, etc. To check, you have seen http://stackoverflow.com/q/2488016/155423? – Shepmaster Jun 10 '16 at 20:31
  • @shepmaster The dylib is linked when `setup.py` is invoked. You can see the clang and gcc invocations which produce the Cython extension in the second and third code blocks. It's definitely the missing -R$ORIGIN invocation that's tripping it up on OSX. – urschrei Jun 10 '16 at 20:48

2 Answers2

1

There are two ways to fix this without using install_name_tool.
Assuming that your dylib is in the same directory as your Cython extension (In this case, cutil.so)

  • Compile your Rust binary with rpath support, and correctly set the rpath in your setup.py Extension instance:
  • Invoke RUSTFLAGS="-C rpath" cargo build --release.
  • The less flexible way is to compile your Rust binary and pass the path in via linker flags. This bakes in the path (albeit with dynamic refs), however:
    • cargo rustc --release -- -C link-args=-Wl,-install_name,@rpath/libname.dylib

After you've enabled rpath support in your Rust dylib, edit your setup.py:

Set (or add) extra_link_args=["-Wl,-rpath", "-Wl,@loader_path/"] in your Extension instance.

You'll know it's worked if you compile your Cython extension (e.g. using setup.py build_ext --inplace), then run otool -l on the resulting .so:

   @rpath/liblonlat_bng.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

If you want your Rust dylib elsewhere relative to your Cython extension, you'll need to adjust your @rpath/ or @loader_path/ paths in setup.py accordingly.

Note: The foregoing works for relative paths on OS X only. On Linux, you can set extra_link_args=["-Wl,-rpath", "-Wl,$ORIGIN"] in your setup.py Extension instance (and adjust it accordingly if your Rust dylib is elsewhere, relative to your extension.)

urschrei
  • 25,123
  • 12
  • 43
  • 84
1

I had the same problem and I believe it is a bug in Distutils. I've made a pull request to fix this and I've also proposed a workaround. See https://github.com/python/cpython/pull/12418

Can you try the following in your setup.py?

from Cython.Distutils.build_ext import new_build_ext as build_ext
# alternative:
# from distutils.command import build_ext

class my_build_ext(build_ext):
    """Workaround for rpath bug in distutils for OSX."""

    def finalize_options(self):
        super().finalize_options()
        # Special treatment of rpath in case of OSX, to work around python
        # distutils bug 36353. This constructs proper rpath arguments for clang.
        # See https://bugs.python.org/issue36353
        if sys.platform[:6] == "darwin":
            for path in self.rpath:
                for ext in self.extensions:
                    ext.extra_link_args.append("-Wl,-rpath," + path)
            self.rpath[:] = []

setup(
    cmdclass={'build_ext': my_build_ext}
    # ...
)
Toon Verstraelen
  • 447
  • 4
  • 7