How do I diagnose why a shared library is not found?

Question

I am trying to run an app which is linked against libboost_system.so

$ ./app
app: error while loading shared libraries: libboost_system.so.1.63.0: cannot open shared object file: \
     No such file or directory

If I run ldd on the app, it shows it can't find the library:

$ ldd ./app
    linux-vdso.so.1 (0x00007ffdedb94000)
    libboost_system.so.1.63.0 => not found
    libpcap.so.1 => /lib64/libpcap.so.1 (0x00007f19a1a7b000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f19a1a70000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f19a1a69000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f19a1879000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f19a1733000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f19a1716000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f19a16f4000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f19a152a000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f19a1ae5000)

The library is present on my system:

$ locate libboost_system.so.1.63.0
/usr/local/lib/libboost_system.so.1.63.0

$ ls -la /usr/local/lib/libboost_system*
-rw-r--r--. 1 root root 47014 Jul  2 16:57 /usr/local/lib/libboost_system.a
lrwxrwxrwx. 1 root root    25 Jul  2 16:57 /usr/local/lib/libboost_system.so -> libboost_system.so.1.63.0
-rwxr-xr-x. 1 root root 19816 Jul  2 16:57 /usr/local/lib/libboost_system.so.1.63.0

ld is configured with /usr/local/lib:

$ ld --verbose | grep SEARCH_DIR | sed 's/; /\n/g'
SEARCH_DIR("=/usr/x86_64-redhat-linux/lib64")
SEARCH_DIR("=/usr/lib64")
SEARCH_DIR("=/usr/local/lib64")
SEARCH_DIR("=/lib64")
SEARCH_DIR("=/usr/x86_64-redhat-linux/lib")
SEARCH_DIR("=/usr/local/lib")                    <----- here
SEARCH_DIR("=/lib")
SEARCH_DIR("=/usr/lib")

I've also run ldconfig to refresh the cache, but that doesn't help.

I tried searching for RPATH or RUNPATH with readelf, neither are specified:

$ readelf -d ./app | grep -i path
< no results >

If I explicitly set LD_LIBARY_PATH then the library is found:

$ LD_LIBRARY_PATH=/usr/local/lib ldd ./app
    linux-vdso.so.1 (0x00007fffa9a5d000)
    libboost_system.so.1.63.0 => /usr/local/lib/libboost_system.so.1.63.0 (0x00007f5fff664000)
    libpcap.so.1 => /lib64/libpcap.so.1 (0x00007f5fff5fc000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f5fff5f1000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f5fff5ea000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f5fff3fa000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f5fff2b4000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f5fff297000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5fff275000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f5fff0ab000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f5fff66b000)

Why can't ld find libboost_system.so.1.63.0 without specifying LD_LIBRARY_PATH?

Not really an idea, but have you checked that you did not link with some '-L' options? May you have some version conflicts cause of other boost libs installed "somewhere" or is /usr/local/lib not searched as /usr/local/lib64 is in place? ld is static linker so please take a look in /etc/ld.so.conf.d/ for the search path of ldd/dynamic loader. — Klaus, Jul 10 '20 at 17:25

score 4 · Accepted Answer · answered Jul 10 '20 at 17:23

4

You are looking at the "wrong" linker. ld is the static linker and isn't what's used by at runtime to locate the shared libraries.

The dynamic linker, ld.so, is what locates shared libraries.

The dynamic linker does look at the LD_LIBRARY_PATH and that's why it works if you set it.

If you want to update the search locations for dynamic linker (and not want to set LD_LIBRARY_PATH), you can create a file in /etc/ld.so.conf.d/ and update /etc/ld.so.cache with ldconfig.

answered Jul 10 '20 at 17:23

P.P

117,907
20
175
238

Ah - TIL! Thank you! – Steve Lorimer Jul 10 '20 at 17:25
Out of interest - do you know why `/usr/local/lib` is **not** configured as a search path by default? – Steve Lorimer Jul 10 '20 at 17:28
ld.so looks at LD_LIBRARY_PATH, /etc/ld.so.conf, /lib and /usr/lib (in that order). Typically, system libs are installed at either /lib or /usr/lib but your boost libs are located at /usr/local/lib - it depends on how you installed boost (manual install?). – P.P Jul 10 '20 at 17:45
Yes I built from source and installed manually. The [FHS spec](https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s09.html) suggests `/usr/local` is a standard directory, so I would have expected it to be included by default. Do you know if not doing so is a recent change adopted by some distros? – Steve Lorimer Jul 10 '20 at 18:37
Seems red-hat related distros don't and debian based do (see [here](https://unix.stackexchange.com/a/357867/28680) and [here](https://stackoverflow.com/questions/4743233/is-usr-local-lib-searched-for-shared-libraries/17653893#17653893)) – Steve Lorimer Jul 10 '20 at 18:42
Yes, `/usr/local` (or `/opt` in case of user apps) is the recommended location for libs maintained by admins. So that's the right approach. – P.P Jul 10 '20 at 18:46
As to why `/usr/local/lib` isn't in the default search paths, as you found, Debian-based systems (checked my mine) do include `/usr/local/lib` (in `libc.conf`). But I am more inclined to think that this isn't a good idea for the reasons mentioned by that [answer](https://unix.stackexchange.com/a/357867). So it's probably best to use LD_LIBRARY_PATH or rpath in your app, instead of updating system-wide ldconfig (this is my opinion of course). – P.P Jul 10 '20 at 18:50
Thanks for the help! – Steve Lorimer Jul 10 '20 at 19:13

Demi-Lune · Answer 2 · 2023-01-06T16:34:39.893

The diagnostic is activated by LD_DEBUG (cf. man ld.so):

LD_DEBUG=libs ./a.out

It shows the search paths that the loader tried: ($LD_LIBRARY_PATH, R[UN]PATH from the app or library's ELF, ld.so.cache). Its output is a little long to copy here, but would look like this:

       265:     find library=libboost_system.so.1.63.0 [0]; searching
       265:      search path= [...]  (LD_LIBRARY_PATH)
                   [trying many files...]
       265:      search path= [...]  (RUNPATH from file ./app)
                   [still trying many files...]
       265:      search cache=/etc/ld.so.cache
                   [still trying many files...]

In your case, you'd probably have seen that /usr/local/lib was not in the cache. Depending on the distribution, it may look by default in /usr/local/lib/x86_64-linux-gnu or /usr/local/lib64 or none at all.

How do I diagnose why a shared library is not found?

2 Answers2