How do I update my libthread_db shared library so as to match libpthread shared library?

Question

I am using CentOS. I didn't set up the server and the environment is quite old.

I tried to debug a multithreaded server program and some error showed up.

Starting program: ./battle
Trying host libthread_db library: libthread_db.so.1.
Host libthread_db.so.1 resolved to: /lib64/libthread_db.so.1.
td_ta_new failed: application not linked with libthread
thread_db_load_search returning 0
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
Trying host libthread_db library: libthread_db.so.1.
Host libthread_db.so.1 resolved to: /lib64/libthread_db.so.1.
td_ta_new failed: versions of libpthread and libthread_db do not match
Trying host libthread_db library: /lib64/libthread_db.so.1.
td_ta_new failed: versions of libpthread and libthread_db do not match
thread_db_load_search returning 0
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

Then I realize that the problem is that the version of libpthread shared library I link doesn't match the version of libthread_db shared library which gdb uses when debugging.

I read these 2 related questions Unable to Debug Multi-Threaded Application with gdb, GDB debugging warnings .

Then I tried to figure out what version of each is by using file command (Please tell me if using file to check the differences is correct. Anyway, I found some differences.)

ldd battle
    linux-vdso.so.1 (0x00007fffcafff000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f390c675000)
    .........

ls -l libpthread*
    -rwxr-xr-x 1 root root 143280 Apr  9  2019 libpthread-2.12.so
    -rwxr-xr-x 1 root root 806517 Dec  2  2018 libpthread-2.17.so
    lrwxrwxrwx 1 root root     18 Jul 29 07:35 libpthread.so.0 -> libpthread-2.17.so

ls -l libthread_db*
    -rwxr-xr-x 1 root root 34488 Apr  9  2019 libthread_db-1.0.so
    lrwxrwxrwx 1 root root    19 Jul 29 07:35 libthread_db.so.1 -> libthread_db-1.0.so

file libpthread-2.17.so 
    libpthread-2.17.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, not stripped

file libthread_db-1.0.so 
    libthread_db-1.0.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped

file libpthread-2.12.so 
    libpthread-2.12.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped

So my progam is linked to libpthread-2.17.so , which is for for GNU/Linux 2.6.16, and gdb uses libthread_db-1.0.so, which is for for GNU/Linux 2.6.18, mismatch? (Well I am not sure if I can compare them based on this information, can I?)

Does it mean that now I need to download a libthread_db-1.0.so which is for for GNU/Linux 2.6.16 and link libthread_db.so.1 to it? Where can I download it from?

By the way, I also feel strange that for libpthread-2.17.so and libpthread-2.12.so on my server , 2.17 > 2.12, the former should be a newer version. But it's for a lower Linux version (2.6.16), maybe someone just copy it and rename the file? I don't know. :(

unfortunately, updating libs in CentOS is a pita and potentially system-bricking.. what you can do is download the sources for thread_db (or whatever it is called) and compile it yourself.. and install it elsewhere. then add the new lib path to LD_LIBRARY_PATH — ewokx, Aug 07 '20 at 08:29
@ewong Can't I just download a `libthread_db.so.1` which is for `for GNU/Linux 2.6.16`, and make a symbol link `libthread_db.so.1 -> the one I download.so` ? — Rick, Aug 07 '20 at 08:31
assuming it is suitable for your system, I believe so. I just normally build it and install the libs elsewhere.. sorry for not being able to help — ewokx, Aug 07 '20 at 08:34
@Rick This [blog post](http://timetobleed.com/notes-about-an-odd-esoteric-yet-incredibly-useful-library-libthread_db/) might be of help — Don Foumare, Aug 07 '20 at 08:40

Employed Russian · Accepted Answer · 2020-08-12T16:45:25.133

Your question is the exact duplicate of the two answers you've already found: someone installed GLIBC-2.17 onto your system (which likely had GLIBC-2.12 originally), but failed to install matching libthread_db.so.1.

The output of file -- GNU/Linux 2.6.18 is a red herring and is irrelevant.

The good news is that updating libpthread_db is risk-free -- if you screw up, the worst that can happen is that GDB will not work for multi-threaded programs, but since that is the current state, you can't make it any worse.

But you should be able to make it better:

download GLIBC-2.17 source, configure and build it (but don't make install, or you'll likely brick your system).
as part of the build, there should now exist build/nptl_db/libthread_db.so.1
test it by pointing GDB to it with set libthread-db-search-path /path/to/build/nptl_db and trying to debug your program again.

If that works, you can overwrite the non-working /lib64/libthread_db.so.1 with the one you just built.

Update:

Is there any command I can use to grab some version information of these shared library files?

For libpthread*.so.0, this is quite easy:

gdb -q /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...

(gdb) x/s nptl_version
0x16037 <nptl_version>: "2.30"

For libthread_db.so.1 it's quite a bit harder: you need to disassemble td_ta_new, and look for memcmp call, which could be inlined:

gdb -q /lib/x86_64-linux-gnu/libthread_db.so.1
Reading symbols from /lib/x86_64-linux-gnu/libthread_db.so.1...

(gdb) disas td_ta_new
Dump of assembler code for function td_ta_new:
   0x0000000000002270 <+0>:     push   %r13
   0x0000000000002272 <+2>:     push   %r12
   0x0000000000002274 <+4>:     push   %rbp
   0x0000000000002275 <+5>:     mov    %rsi,%rbp
...
   0x00000000000022d6 <+102>:   callq  0x2110 <ps_pdread@plt>
   0x00000000000022db <+107>:   mov    %eax,%r12d
   0x00000000000022de <+110>:   test   %eax,%eax
   0x00000000000022e0 <+112>:   jne    0x2318 <td_ta_new+168>
   0x00000000000022e2 <+114>:   cmpl   $0x30332e32,0x13(%rsp)
   0x00000000000022ea <+122>:   je     0x2340 <td_ta_new+208>
   0x00000000000022ec <+124>:   mov    $0x16,%r12d
   0x00000000000022f2 <+130>:   mov    0x18(%rsp),%rax
   0x00000000000022f7 <+135>:   xor    %fs:0x28,%rax
   0x0000000000002300 <+144>:   jne    0x2388 <td_ta_new+280>
   0x0000000000002306 <+150>:   add    $0x28,%rsp
   0x000000000000230a <+154>:   mov    %r12d,%eax
   0x000000000000230d <+157>:   pop    %rbx
   0x000000000000230e <+158>:   pop    %rbp
   0x000000000000230f <+159>:   pop    %r12
   0x0000000000002311 <+161>:   pop    %r13
   0x0000000000002313 <+163>:   retq
...

Here, at address 0x022e2 is an inlined call to memcmp("2.30", <data-read-from-nptl_version>, 5).

Thank you so much. I will try your solution later and accept your answer. Btw, since the output of `file` is irrelevant, then how do I know the version of `libthread_db.so.1` and the version of `libpthread-2.17.so`? Is there any command I can use to grab some version information of these shared library files? Because someone could just rename the shared library file in whatever version they like. — Rick, Aug 12 '20 at 07:08
@Rick Unfortunately there is no easy way to to deduce which version of GLIBC `libthread_db.so.1` corresponds to. — Employed Russian, Aug 12 '20 at 16:21

How do I update my libthread_db shared library so as to match libpthread shared library?

1 Answers1