dylib cannot load libstd when compiled in a workspace

Question

I have a project with the following structure:

Cargo.toml
my_script.py
my_lib:
    - Cargo.toml
    - src
my_bin:
    - Cargo.toml
    - src

Where:

my_lib is a Rust library with crate-type = ["dylib"]
my_bin is a Rust binary application using my_lib
my_script.py is a Python 3 script that also uses my_lib

The root Cargo.toml contains a basic workspace declaration:

[workspace]
members = [
    "my_lib",
    "my_bin"
]

Everything works properly if I execute cargo build and cargo run -p my_bin. The problem comes with the Python script.

In this script, I load the my_lib lib file using the following code:

from ctypes import cdll
from sys import platform

if platform == 'darwin':
    prefix = 'lib'
    ext = 'dylib'
elif platform == 'win32':
    prefix = ''
    ext = 'dll'
else:
    prefix = 'lib'
    ext = 'so'

# Working path:
# lib_path = './my_lib/target/debug/{}my_lib.{}'.format(prefix, ext)

# Buggy "Library not loaded: @rpath/libstd-d00eaa6834e55536.dylib" path:
lib_path = './target/debug/{}my_lib.{}'.format(prefix, ext)

lib = cdll.LoadLibrary(lib_path)
my_func = lib.my_func
my_func()

If I use the library file from the library directory (./my_lib/target/...), the script has no problem loading the library and executing its functions.

But if I use the library file from the workspace directory (./target/...) I get the following error when trying to load the library:

OSError: dlopen(./target/debug/libpeglrs.dylib, 6): Library not loaded: @rpath/libstd-d00eaa6834e55536.dylib

In the same fashion, trying to execute my_bin directly from the workspace target directory yields the same error (even though cargo run -p my_bin work flawlessly).

Using the software "Dependency Walker", I found that the my_lib library cannot find the Rust libstd library (has the previous error message explain).

Manually exporting the path that contains the Rust toolchain library into the environment PATH fixes the issue. This is however far from ideal and not portable. I also don't understand why this issue is only occurring when using the workspace target.

So, why can't the workspace target find rust's libstd when each project target can? Is there a way to fix this issue that wouldn't require to find the toolchain path and modify an environment variable?

I found this issue: https://github.com/rust-lang/rustup.rs/issues/932 And running `rustup run beta python3 editor.py` does work, but still doesn't explain why only the workspace is buggy. — Maeln, Mar 21 '19 at 14:22
It's working but I lose the ability to use the lib directly has a dependency in the binary project (since its losing the rust specific info). It would mean that I would have to make ffi against my own Rust lib. Not ideal :/ — Maeln, Mar 21 '19 at 19:26

zrzka · Accepted Answer · 2019-03-29T14:20:17.633

Dynamic linking is not easy sometimes. The error message Library not loaded: @rpath/libstd-d00eaa6834e55536.dylib is pretty clear. You have a problem with DYLD_LIBRARY_PATH (macOS).

TL;DR

Your DYLD_LIBRARY_PATH doesn't contain Rust libraries path. Put following into your ~/.bash_profile:

source "$HOME/.cargo/env"
export RUST_SRC_PATH="$(rustc --print sysroot)/lib/rustlib/src/rust/src"
export DYLD_LIBRARY_PATH="$(rustc --print sysroot)/lib:$DYLD_LIBRARY_PATH"

Explanation

I followed your project structure except one thing - I removed _ (my_bin -> mybin, ...).

`cargo run --bin mybin` vs `target/debug/mybin`

As a first thing, check what otool -L target/debug/mybin says:

target/debug/mybin:
    /Users/robertvojta/Work/bar/target/debug/deps/libmylib.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libstd-d4fbe66ddea5f3ce.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)
    /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)

Notice the @rpath. If you don't know what it is, I'd recommend to read Mike Ash posts:

Also run man dlopen and read the SEARCHING section. It's pretty long to copy & paste it here, so, just the first sentence:

dlopen() searches for a compatible Mach-O file in the directories specified by a set of environment variables and the process's current working directory.

You'll learn about DYLD_LIBRARY_PATH and other environment variables.

In your shell, what's the output of echo $DYLD_LIBRARY_PATH command? I assume it's empty / does not contain Rust libraries path.

Add following lines to your mybin:main.rs ...

println!(
    "DYLD_LIBRARY_PATH={}",
    std::env::var("DYLD_LIBRARY_PATH").unwrap_or("N/A".to_string())
);

... and run cargo run --bin mybin. You should see something like this:

DYLD_LIBRARY_PATH=~/.rustup/toolchains/stable-x86_64-apple-darwin/lib

cargo run injects this environment variable for you.

From where you can get the right value? Run rustc --print sysroot and append /lib to the output.

If you'd like to run mybin directly (without cargo), you can do it in this way:

DYLD_LIBRARY_PATH="$(rustc --print sysroot)/lib:$DYLD_LIBRARY_PATH" target/debug/mybin

Python script

Add similar lines to your run.py script:

import os
print('DYLD_LIBRARY_PATH: {}'.format(os.environ.get('DYLD_LIBRARY_PATH', 'N/A')))

If it prints N/A, DYLD_LIBRARY_PATH is not set. You can fix this in a similar way:

DYLD_LIBRARY_PATH="$(rustc --print sysroot)/lib:$DYLD_LIBRARY_PATH" python run.py

macOS & System Integrity Protection

Be aware that you can't use system Python for this ...

$ echo $DYLD_LIBRARY_PATH
~/.rustup/toolchains/stable-x86_64-apple-darwin/lib:
$ /usr/bin/python run.py
DYLD_LIBRARY_PATH: N/A
Traceback (most recent call last):
  File "./run.py", line 21, in <module>
    lib = cdll.LoadLibrary(lib_path)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 443, in LoadLibrary
    return self._dlltype(name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 365, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(./target/debug/libmylib.dylib, 6): Library not loaded: @rpath/libstd-d4fbe66ddea5f3ce.dylib
  Referenced from: /Users/robertvojta/Work/bar/target/debug/libmylib.dylib
  Reason: image not found

... but you can use one installed via brew for example ...

$ echo $DYLD_LIBRARY_PATH
/Users/robertvojta/.rustup/toolchains/stable-x86_64-apple-darwin/lib:
$ /usr/local/bin/python run.py
DYLD_LIBRARY_PATH: /Users/robertvojta/.rustup/toolchains/stable-x86_64-apple-darwin/lib:

The reason is SIP. SIP was introduced in El Capitan and it can be in your way. You can experience stuff like:

$ env | grep DYLD
$ echo $DYLD_LIBRARY_PATH
/Users/robertvojta/.rustup/toolchains/stable-x86_64-apple-darwin/lib:

Here's the SIP description page. SIP protects folders like /usr, /bin, /sbin, but it doesn't protect /usr/local for example.

What does it mean? SIP does lot of things, but one of them is trashing DYLD_LIBRARY_PATH value. Shebang lines like ...

#!/usr/bin/env python
#!/usr/bin/python

... wont work for you. You have to use Python interpreter, which is not installed in system (& protected) folders. Install one via brew, install Anaconda, ...

SIP can be disabled, but DON'T do this.

Another way how to fix this is to replace @rpath in your mylib with a full path via install_name_tool (man install_name_tool). More info in Why is install_name_tool and otool necessary for Mach-O libraries in Mac Os X?.

Example:

$ otool -L target/debug/mybin
target/debug/mybin:
    /Users/robertvojta/Work/bar/target/debug/deps/libmylib.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libstd-d4fbe66ddea5f3ce.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)
    /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
$ install_name_tool -change @rpath/libstd-d4fbe66ddea5f3ce.dylib /Users/robertvojta/.rustup/toolchains/stable-x86_64-apple-darwin/lib/libstd-d4fbe66ddea5f3ce.dylib target/debug/libmylib.dylib
$ otool -L target/debug/libmylib.dylib
target/debug/libmylib.dylib:
    /Users/robertvojta/Work/bar/target/debug/deps/libmylib.dylib (compatibility version 0.0.0, current version 0.0.0)
    /Users/robertvojta/.rustup/toolchains/stable-x86_64-apple-darwin/lib/libstd-d4fbe66ddea5f3ce.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)
    /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
$ /usr/bin/python run.py
DYLD_LIBRARY_PATH: N/A
Hallo

As you can see, there's no @rpath now, DYLD_LIBRARY_PATH is not set, but it works (Hallo is printed via hallo function from libmylib.dylib) with system Python interpreter.

Be aware of one thing - macOS dynamic libraries behave differently when compared to Linux for example.

If you do not want to mess with it, you can change mylib crate-type to ["rlib", "cdylib"], but that's not what you probably want.