Why does the program which is compiled against the installed glibc not run normally?

Question

Thanks in advance.

my development environment：

$ cat /proc/version
Linux version 5.4.0-66-generic (buildd@lgw01-amd64-016) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #74~18.04.2-Ubuntu SMP Fri Feb 5 11:17:31 UTC 2021

$ ld --version
GNU ld (GNU Binutils for Ubuntu) 2.30
Copyright (C) 2018 Free Software Foundation, Inc.

$ getconf GNU_LIBC_VERSION 
glibc 2.27

$ #my glibc source version is 2.32.9000-development
$ cat ./version.h
/* This file just defines the current version number of libc.  */
#define RELEASE "development"
#define VERSION "2.32.9000"

For some reasons, I need to modify and test glibc. I follow the steps of this website(https://sourceware.org/glibc/wiki/Testing/Builds#Compile_against_glibc_in_an_installed_location) to modify glibc and write test programs.

1. compile glibc.(confgure and make)
2. install glibc.(make install to a directory)
3. ...other steps in the website above.

I successfully modified some pthread functions and passed the test (the test program I wrote can compiled against the install glibc and ran successfully). ldd the program.

$ ldd ./exec/1-1.out 
    linux-vdso.so.1 (0x00007ffcbf367000)
    libpthread.so.0 => /home/cjl-target/gnu/install/lib64/libpthread.so.0 (0x00007fcadcea9000)
    libc.so.6 => /home/cjl-target/gnu/install/lib64/libc.so.6 (0x00007fcadcaed000)
    /home/cjl-target/gnu/install/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fcadd2ca000)

As shown above, the shared libraries that the program depends on all point to the glibc installation path.

But when I compiled message-queue's test program(test mq_unlink) and ran it, failed as bellow:

./exec/1-1.out: symbol lookup error: /lib/x86_64-linux-gnu/libpthread.so.0: undefined symbol: __libc_vfork, version GLIBC_PRIVATE

check the library that is depended by the program:

$ ldd ./exec/1-1.out 
    linux-vdso.so.1 (0x00007ffce3f72000)
    librt.so.1 => /home/cjl-target/gnu/install/lib64/librt.so.1 (0x00007f0a389a2000)
    libc.so.6 => /home/cjl-target/gnu/install/lib64/libc.so.6 (0x00007f0a385e6000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0a383c7000)
    /home/cjl-target/gnu/install/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f0a38dac000)

As shown above, the shared libraries libpthread.so.0 points to the system library. Why?

my compile script is(from the website above):

# dobuild.sh
SYSROOT=/home/xxx/xxx/xxx #the glibc's installation path
(set -x; \
gcc \
  -L${SYSROOT}/usr/lib64 \
  -I${SYSROOT}/usr/include \
  --sysroot=${SYSROOT} \
  -Wl,-rpath=${SYSROOT}/lib64 \
  -Wl,--dynamic-linker=${SYSROOT}/lib64/ld-linux-x86-64.so.2 \
  -Wall $*
)

when I compile the pthread's test program:./dobuild 1-1.c -pthread -Wall

when I compile the mq's test program:./dobuild 1-1.c -lrt -Wall

In addition, it is confusing that when invoke the pthread_create in the mq_unlink's test program, compiling it ./dobuild 1-1.c -lrt -pthread, the ldd result shows that all dependent libraries point to the installed glibc.

I've tried multiple variations of this, but none of them seem to work. Any ideas?

Answer 1

First, you should stop using ldd -- in the presence of multiple GLIBCs on a host, ldd is more likely to mislead than to illuminate.

If you want to see which libraries are really loaded, do this instead:

LD_TRACE_LOADED_OBJECTS=1 ./exec/1-1.out

Second, you should almost never use $* in shell scripts. Use "$@" instead (note: quotes are important). See this answer.

Third, the behavior you are observing is easily explained. To understand it, you need to know the difference between DT_RPATH and DT_RUNPATH, described here.

You can verify that your binaries are currently using RUNPATH, like so:

readelf -d 1-1.out | grep 'R.*PATH'

And you can verify that everything starts working as you expect by adding -Wl,--disable-new-dtags to the link command (which would cause the binary to use RPATH instead).

To summarize:

• RUNPATH affects the search for the binary itself, but not for any libraries the binary depends on.
• RPATH affects the search path for the binary and all libraries it depends on.
• with RUNPATH, expected libpthread.so.0 is found only when the binary depends on it directly, but not when the dependency on libpthread is indirect (via librt).
• with RPATH, expected libpthread.so.0 is found regardless of whether the dependency is direct or indirect.

Update:

If I want to use DT_RUNPATH, how to set the library runpath for librt?

You would need to link librt.so with -rpath=${SYSROOT}/lib64.

You could edit the rt/Makefile, or build with:

make LDFLAGS-rt.so='-Wl,--enable-new-dtags,-z,nodelete,-rpath=${SYSROOT}/lib64'

You would need to do the same for any other library that may bring transitive dependency on other parts of GLIBC. I don't know of a general way to do this, but setitng LDFLAGS-lib.so='-Wl,-rpath=${SYSROOT}/lib64' and rebuilding everything might do the trick.