Getting executables to run on multiple Linux OS's:

Question

I'm attempting to get a very basic program compiled and run on multiple OS's. The program just attempts to print it's filename to stream, using boost::filesystem, so that I can verify that loading .so's works as expected.

I compile it on an Ubuntu box:

$ uname -a
Linux ubuntu 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

And I have a CentOS box which I attempt to run it on:

$ uname -a
Linux localhost.localdomain 3.10.0-123.20.1.el7.x86_64 #1 SMP Thu Jan 29 18:05:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I compile the executable using $ORIGIN so that the linked boost libraries will be picked up from my directory, then I ldd the boost libraries and cp them into the same. So, the lib directory looks as follows:

deliverable/
deliverable/hello
deliverable/.lib/
deliverable/.lib/libc.so.6
deliverable/.lib/libboost_system.so.1.58.0
deliverable/.lib/libpthread.so.0
deliverable/.lib/libm.so.6
deliverable/.lib/libstdc++.so.6
deliverable/.lib/libboost_filesystem.so.1.58.0
deliverable/.lib/libboost_filesystem.so
deliverable/.lib/libfoo.so
deliverable/.lib/libboost_system.so
deliverable/.lib/libgcc_s.so.1

where hello is the executable I want to run. However, on the CentOs box, I get the following error:

$ ./hello
$ ./hello: relocation error: ~/deliverable/.lib/libc.so.6: symbol _dl_find_dso_for_object, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference

How can this be fixed? I'd also like to know if this pattern violates best practices regarding shipping compiled code between Linux machines . . .

More info if relevant:

$ cat Makefile 
CXX = g++
CPPFLAGS := -Wall -g -Wfatal-errors -std=c++11 -I./inc -fPIC
DELIVERABLE = $(CURDIR)/deliverable
LIB = $(DELIVERABLE)/.lib

all: $(DELIVERABLE)/hello

$(DELIVERABLE)/hello: main.o $(LIB)/libfoo.so
    $(CXX) -L./deliverable/.lib -Wl,--allow-shlib-undefined -Wl,-rpath='$$ORIGIN/.lib' -o $@ $< -lfoo

main.o: main.cc
    $(CXX) $(CPPFLAGS) -c $< -o $@

$(LIB)/libfoo.so: foo.o
    $(CXX) -L./deliverable/.lib -Wl,--allow-shlib-undefined -Wl,-rpath='$$ORIGIN/.lib' -shared -o $@ $^ -lboost_system -lboost_filesystem

foo.o: foo.cc
    $(CXX) $(CPPFLAGS) -c $< -o $@

clean:
    rm -f *.o $(LIB)/libfoo.so $(DELIVERABLE)/hello

$ cat main.cc 
#include "foo.hh"

int main()
{
    hello();
}
$ cat foo.hh 
#ifndef FOO_HH
#define FOO_HH

void hello();

#endif
$ cat foo.cc
#include "foo.hh"
#include <boost/filesystem.hpp>
#include <iostream>

void hello()
{
    boost::filesystem::path p{__FILE__};
    std::cout << "p.parent_path() " << p.parent_path() << '\n';
    std::cout << "p.string()      " << p.string() << '\n';
    std::cout << "__FILE__        " << __FILE__ << '\n';
}

I also tried this on a RHEL box, which gave an even worse error:

$ uname -a
Linux localhost.localdomain 2.6.18-164.6.1.el5 #1 SMP Tue Nov 3 ... EXT 2009 x86_64 x86_64 GNU/Linux

Running it on this machine crashed with:

$./hello
./hello: error while loading shared libraries: ~/deliverable/.lib/libm.so.6: unexpected PLT reloc type 0x25

Is there a reason why you also included libc, libstdc++, libm, libgcc, etc. to the deliverables directory? As far as I can see you only need `libfoo` and `libboost_system` for your application. See this [thread about how to deploy a custom libc](http://stackoverflow.com/questions/10763394/how-to-build-a-c-program-using-a-custom-version-of-glibc). It appears to me your problem is the your linker and the dynamic loader are not understanding each other (relocation errors, PLT errors and the like) correctly. — djf, Apr 09 '15 at 18:26
I deployed only `libfoo` and `libboost_system`, but then the `libstdc++` and so on weren't found on the other machine. (They were there, but had the wrong versions.) — user14717, Apr 09 '15 at 18:30
ah, alright. Can you also post the error message you get if you're not deploying the additional libs — djf, Apr 09 '15 at 18:47
On RHEL: `./hello: /lib64/libc.so.6: version GLIBC_2.14 not found (required by ~/deliverable/.lib/libboost_filesystem.so.1.58.0)` — user14717, Apr 09 '15 at 19:44
Can tell us the output of `g++ --version` for all your relevant OSses? — nils, Apr 09 '15 at 19:51
@nils: The goal is to make it so that we can deliver compiled software on machines without developer tools. So if `g++ --version` is a relevant consideration, then I need a new delivery pattern. — user14717, Apr 09 '15 at 20:50
In that case - and in order to be on the safe side - you will have to ship **all** dependencies; recursively call `ldd` on each dependency to find all. On the target platforms make sure that these are found *before* the corresponding system libs. — nils, Apr 09 '15 at 21:08
@Nick I've found an interesting looking blog post on [how to compile portable Linux binaries](http://insanecoding.blogspot.in/2012/07/creating-portable-linux-binaries.html). Also have a look at this SO thread: [Running a C program compiled here causes a GLIBC library not found error on another server - is it my fault or theirs?](http://stackoverflow.com/questions/10594262/running-a-c-program-compiled-here-causes-a-glibc-library-not-found-error-on-anot) — djf, Apr 09 '15 at 21:48
@Nick if you decide to really deploy all shared library dependencies including libc, you also have to specify a custom loader. See [this thread](http://stackoverflow.com/questions/847179/multiple-glibc-libraries-on-a-single-host). I tried that once in order to get an application running on a prehistoric legacy system... — djf, Apr 09 '15 at 22:09
@nils: I did recursively call `ldd`, and the `strace` output shows that the right libraries are being loaded. — user14717, Apr 09 '15 at 23:41
@Nick You're welcome. After sleeping over the issue I would try to statically build and link to boost. If boost_filesystem.so has dependencies that are not present on the target machine, then surely boost_filesystem.a should work around that right... right? Please post an answer if you crack this problem. — djf, Apr 10 '15 at 06:15
@NickThompson You mean on your CentOS machine they are loaded, right? That may be so, but still since it's a different version of the GCC on that machine (I belief it's an older one) the library `ld-linux-x86-64.so.2` does not (yet) have the symbols that were assumed to be available on your Ubuntu machine. That is why I said you need to copy all libraries listed by `ldd` on Ubuntu machine and copy them to your CentOS machine. — nils, Apr 10 '15 at 08:09
@djf: I'm not sure that we have legal rights to link all our dependencies statically (but of course boost we can). I'm thinking now that I'll just spin up a virtualbox for every supported environment and do a build there. Shipping a loader is insane! — user14717, Apr 10 '15 at 13:35

score 0 · Answer 1 · answered Apr 09 '15 at 17:34

I thought, after a system has additional .so files installed, you have to 'tell' ld they are there?

ldconfig is the command I am thinking of:

ldconfig creates the necessary links and cache (for use by the run-time linker, ld.so) to the most recent shared libraries found in the directories specified on the command line, in the file /etc/ld.so.conf, and in the trusted directories (/usr/lib and /lib). ldconfig checks the header and file names of the libraries it encounters when determining which versions should have their links updated. ldconfig ignores symbolic links when scanning for libraries.

http://linux.about.com/od/commands/l/blcmdl8_ldconfi.htm

Use of the '$ORIGIN' variable takes care of this. In addition, `strace ./hello` shows that the proper libraries are being loading. — user14717, Apr 09 '15 at 18:09

score 0 · Answer 2 · answered Apr 09 '15 at 18:40

I guess this is a typical case of subtle differences between Linux distributions:

Ubuntu which is based on Debian and CentOS which is based on RedHat are known to use different locations to store system libraries and - to make things worse - also use different naming schemes.

And even more complications can come up if the age of the distributions varies significantly (new Ubuntu release vs. old CentOS or vice versa).

You compiled all your libraries on your Ubuntu machine. Running ldd ~/deliverable/.lib/libc.so.6 will show you a complete list of dependencies, i.e. all dependencies are found.

I am almost certain that running that same command on your CentOS machine will show you an incomplete list (one or more dependencies cannot be found). This is most likely due to the difference between Linux distributions mentioned above.

From the top of my head I see the following ways to deal with this situation:

on your CentOS machine identify distribution-specific location and name of the missing dependencies and sym-link them using the same absolute paths that were shown by ldd on your Ubuntu machine. This is a quick and dirty solution which may or (rather) may not work. I would advise against it.
Compile all the libs in deliverable/.lib/* for the specific target platform and link them accordingly. That would also put away with the necessity to ship libstd++ since it will be linked to the right one.

Just ran an `ldd` on the CentOS. All dependencies are found. — user14717, Apr 09 '15 at 18:44
@nils It's always the same plattform, i.e. x64. How/why would you compile any differently for CentOS or Ubuntu? — djf, Apr 09 '15 at 18:46
@djf I implicitly assumed identical hardware for all Linux OS's, so x64-or-not isn't the problem. The current deployment assumes ABI-compatibility between the Linux distributions, which cannot be guaranteed. Building the applications/libraries separately for each Linux distribution ensures correct linking and name mangling of symbols for that specific distribution. — nils, Apr 09 '15 at 20:07
@NickThompson Ok, that rules out that problem. Can you check your version(s) of `g++`? If they differ too much it could explain the missing symbols on CentOS. — nils, Apr 09 '15 at 20:09
@nils You got a point there. I naively thought distros were compatible :) — djf, Apr 09 '15 at 21:00

Getting executables to run on multiple Linux OS's:

2 Answers2