7

I build an R package which uses Rcpp and links to a third-party shared object (libbarraopt.so) (which also links to other shared objects such as liboptsrvr.so in its own directory). To ensure that it is able to find those shared objects it links against, I put the following variables in ~/.Renviron:

BARRA_OPS_HOME=${HOME}/bin/BarraOptimizer8.5

In the package, I create the following src/Makevars:

BARRA_LIB=$(BARRA_OPS_HOME)/lib/intel64
BARRA_INCLUDE=$(BARRA_OPS_HOME)/include
PKG_CXXFLAGS=-I$(BARRA_INCLUDE)
PKG_CFLAGS=-I$(BARRA_INCLUDE)
PKG_LIBS=-L$(BARRA_LIB) -Wl,-R,$(BARRA_LIB) -lbarraopt

Under Ubuntu 16.04, I can build, load, and use the package successfully without any problem. However, when I test exactly the same package when OS is upgraded to 17.10, the package can be built but it cannot be loaded, saying:

g++ -std=gnu++11 -I/usr/share/R/include -DNDEBUG  -I"/home/renkun/R/x86_64-pc-linux-gnu-library/3.4/Rcpp/include"   -I/home/renkun/bin/BarraOptimizer8.5/include -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c RcppExports.cpp -o RcppExports.o
** libs
g++ -std=gnu++11 -I/usr/share/R/include -DNDEBUG  -I"/home/renkun/R/x86_64-pc-linux-gnu-library/3.4/Rcpp/include"   -I/home/renkun/bin/BarraOptimizer8.5/include -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c barraopt.cpp -o barraopt.o
g++ -std=gnu++11 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o barraopt.so RcppExports.o barraopt.o -L/home/renkun/bin/BarraOptimizer8.5/lib/intel64 -Wl,-R,/home/renkun/bin/BarraOptimizer8.5/lib/intel64 -lbarraopt -L/usr/lib/R/lib -lR
installing to /tmp/Rtmpvbb6Io/devtools_install_42a342a07f84/barraopt/libs
* DONE (barraopt)
Error in dyn.load(dllfile) : 
  unable to load shared object '/home/renkun/Workspaces/barraopt/src/barraopt.so':
  liboptsrvr.so: cannot open shared object file: No such file or directory
Calls: suppressPackageStartupMessages ... <Anonymous> -> load_all -> load_dll -> library.dynam2 -> dyn.load
Execution halted

Exited with status 1.

It seems that -Wl,-rpath is not effective here.

Under a machine with Ubuntu 16.04, ldd src/barraopt.so shows that all dynamic linking is corrected resolved. (BARRA_OPS_HOME = /home/ken/bin/BarraOptimizer8.5)

linux-vdso.so.1 =>  (0x00007ffc89a16000)
libbarraopt.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libbarraopt.so (0x00007f85dae49000)
libimf.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libimf.so (0x00007f85da97f000)
libR.so => /usr/lib/libR.so (0x00007f85da346000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f85d9fc4000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f85d9dae000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f85d99e3000)
liboptsrvr.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/liboptsrvr.so (0x00007f85d7b10000)
libopsproto.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libopsproto.so (0x00007f85d77a1000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f85d7497000)
libintlc.so.5 => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libintlc.so.5 (0x00007f85d7249000)
libblas.so.3 => /usr/lib/libblas.so.3 (0x00007f85d6fe8000)
libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007f85d6da1000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f85d6b31000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f85d690f000)
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f85d66fe000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f85d64e4000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f85d62dc000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f85d60d7000)
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f85d5eb5000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f85d5c98000)
/lib64/ld-linux-x86-64.so.2 (0x000055fb75088000)
libifcore.so.5 => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libifcore.so.5 (0x00007f85d5961000)
libifport.so.5 => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libifport.so.5 (0x00007f85d5732000)
libsvml.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libsvml.so (0x00007f85d4e6d000)
libmosek64.so.7.0 => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libmosek64.so.7.0 (0x00007f85d3c63000)
libiomp5.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libiomp5.so (0x00007f85d396b000)
libprotobuf.so.6 => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libprotobuf.so.6 (0x00007f85d3668000)
libbridge_common.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libbridge_common.so (0x00007f85d3417000)
libsharc_xmlxproto.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libsharc_xmlxproto.so (0x00007f85d31a4000)
libboost_thread.so.1.49.0 => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libboost_thread.so.1.49.0 (0x00007f85d2f8a000)
libopenblas.so.0 => /usr/lib/libopenblas.so.0 (0x00007f85d0ef5000)
libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f85d0ccc000)
libxerces-c-3.1.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libxerces-c-3.1.so (0x00007f85d07c4000)
libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f85d0499000)
libnsl.so.1 => /lib/x86_64-linux-gnu/libnsl.so.1 (0x00007f85d027f000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f85d0040000)

However, with the same source, under Ubuntu 17.10, ldd shows that the shared objects libbarraopt.so links against are not resolved even though -Wl,-rpath is secified: (BARRA_OPS_HOME = /home/renkun/bin/BarraOptimizer8.5)

linux-vdso.so.1 =>  (0x00007ffe067f5000)
libbarraopt.so => /home/renkun/bin/BarraOptimizer8.5/lib/intel64/libbarraopt.so (0x00007f3dc5f0c000)
libR.so => /usr/lib/libR.so (0x00007f3dc58e4000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f3dc555e000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3dc5208000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f3dc4ff1000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3dc4c11000)
liboptsrvr.so => not found
libopsproto.so => not found
libblas.so.3 => /usr/lib/x86_64-linux-gnu/libblas.so.3 (0x00007f3dc49b6000)
libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007f3dc4770000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f3dc44fe000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f3dc42d8000)
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f3dc40c8000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f3dc3eab000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f3dc3ca3000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3dc3a9f000)
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f3dc3870000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3dc3651000)
/lib64/ld-linux-x86-64.so.2 (0x00007f3dc6526000)
libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x00007f3dc13ab000)
libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f3dc1182000)
libgfortran.so.4 => /usr/lib/x86_64-linux-gnu/libgfortran.so.4 (0x00007f3dc0da3000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f3dc0b63000)

It looks like only libbarraopt.so is linked with the correct path but shared objects it links against are missing.

I'm wondering what might be wrong with my build configurations that breaks under the toolchain shipped by 17.10. Although using global config such as ldconfig would solve such problem, I prefer not because some .so it relies have conflict with the version the OS ships. I'd rather use a locally configured version without affecting the global config.

Kun Ren
  • 4,715
  • 3
  • 35
  • 50
  • Well just got home ... but let me assure you that just this week I set a new project up with `-rpath` under 17.04 and it worked as expected... A more minimal example may reduce the barrier to helping you here. – Dirk Eddelbuettel Nov 05 '17 at 02:14

1 Answers1

13

It seems that -Wl,-rpath is not effective here.

What likely happened is that your updated linker emits DT_RUNPATH dynamic tag, where the old linker emitted DT_RPATH. (It's also possible that your old linker was GNU-ld, and the new one is Gold.)

The DT_RUNPATH is preferred as more correct, and affects search path of the binary itself, but not of any of the dependent libraries.

The DT_RPATH has global effect, similar to adding the directory to LD_LIBRARY_PATH environment variable.

You can verify this with: readelf -d a.out | grep 'R.*PATH'.

If you do see the RPATH vs. RUNPATH difference, and in fact are using Gold, you can force the "old" behavior with -Wl,--disable-new-dtags (GNU ld also had --disable-new-dtags added to it recently, so it should work for either linker).

Employed Russian
  • 199,314
  • 34
  • 295
  • 362
  • Yes, `RUNPATH` instead of `RPATH` appear in ELF. In my case, `RPATH` is what I need. – Kun Ren Nov 12 '17 at 00:13
  • Looks like the default behavior of `ld` shipped with Ubuntu 16.04 is different from that with Ubuntu 17.10. I look at both locations that `/usr/bin/ld` points to, they both point to `x86_64-linux-gnu-ld` but with different version. Is one `GNU-ld` and the other `Gold` or just version breaking? – Kun Ren Nov 12 '17 at 03:37
  • @KunRen I don't have easy access to either `16.04` of `17.10`. You can run `ld --version`. GNU-ld will look like this: `GNU ld (GNU Binutils for Ubuntu) ...`. Gold will look like this: `GNU gold (GNU Binutils for Ubuntu ...)` – Employed Russian Nov 12 '17 at 03:47
  • Ubuntu 16.04: GNU ld (GNU Binutils for Ubuntu) 2.26.1 Ubuntu 17.10: GNU ld (GNU Binutils for Ubuntu) 2.29.1 – Kun Ren Nov 12 '17 at 03:49
  • @KunRen It's also possible that GCC invokes something other than `/usr/bin/ld` (i.e. GCC on `17.10` may be configured to invoke Gold). Use `gcc --print-prog-name=ld` to find out. – Employed Russian Nov 12 '17 at 04:08
  • @EmployedRussian you closed my question as a dup of this. I don't get *any* results from `readelf` for `DT_R.*PATH` (I do get `RUNPATH` by itself) – Steve Lorimer Dec 12 '17 at 14:54
  • @SteveLorimer Right. `DT_` is not part of the output. Answer corrected. – Employed Russian Dec 12 '17 at 15:02
  • @EmployedRussian thanks! I have tested on both Ubuntu 16.04 and 17.10 and `-Wl,--disable-new-dtags` flag has fixed my problem. – Steve Lorimer Dec 12 '17 at 15:08
  • 1
    @EmployedRussian the comments at the bottom of [this blog post](http://blog.qt.io/blog/2011/10/28/rpath-and-runpath/) mention **`DT_RPATH` is considered a flaw in the ELF specification**. How does one propagate the `RUNPATH` to transitive dependencies? Is the *"fix"* to set `RUNPATH` on the dependencies themselves? Is there a better way? TIA. – Steve Lorimer Dec 12 '17 at 15:15
  • 1
    @SteveLorimer Yes: the idea is that every ELF image should be built to correctly describe its own required runtime paths and its own dependencies, rather than depending on which "main" application it is loaded into. Setting `RUNPATH` on each of the transitive dependencies is a way to achieve that. – Employed Russian Dec 12 '17 at 15:24
  • How can I get ld to embed RPATH and not RUNPATH? This would be a quick solution for a similar issue I'm dealing with until I can get around to migrate all my build to the C++ 11 linkage convention and eliminate much of the complexity. – Isac Casapu Aug 22 '18 at 08:36
  • @EmployedRussian - I couldn't find any suggestion as to how can I get ld to embed RPATH and not RUNPATH. I might be missing something here, but I'd appreciate a clearer pointer. – Isac Casapu Aug 23 '18 at 12:07
  • 1
    @YitzikC Use `-Wl,--disable-new-dtags` flag. – Employed Russian Aug 23 '18 at 14:20