0

Although Linux doesn't provide pstack as Solaris does, RedHat provides a script can do the same thing: 

#!/bin/bash

if test $# -ne 1; then
    echo "Usage: `basename $0 .sh` <process-id>" 1>&2
    exit 1
fi

if test ! -r /proc/$1; then
    echo "Process $1 not found." 1>&2
    exit 1
fi

# GDB doesn't allow "thread apply all bt" when the process isn't
# threaded; need to peek at the process to determine if that or the
# simpler "bt" should be used.

backtrace="bt"
if test -d /proc/$1/task ; then
    # Newer kernel; has a task/ directory.
    if test `/bin/ls /proc/$1/task | /usr/bin/wc -l` -gt 1 2>/dev/null ; then
        backtrace="thread apply all bt"
    fi
elif test -f /proc/$1/maps ; then
    # Older kernel; go by it loading libpthread.
    if /bin/grep -e libpthread /proc/$1/maps > /dev/null 2>&1 ; then
        backtrace="thread apply all bt"
    fi
fi

GDB=${GDB:-/usr/bin/gdb}

if $GDB -nx --quiet --batch --readnever > /dev/null 2>&1; then
    readnever=--readnever
else
    readnever=
fi

# Run GDB, strip out unwanted noise.
$GDB --quiet $readnever -nx /proc/$1/exe $1 <<EOF 2>&1 |
$backtrace
EOF
/bin/sed -n \
    -e 's/^(gdb) //' \
    -e '/^#/p' \
    -e '/^Thread/p'

Executing the script in Suse: 

linux:~ # pstack 7286
Thread 3 (Thread 0x7f7c074b5700 (LWP 7287)):
#0  0x00007f7c055f7a9d in read () from /lib64/libpthread.so.0
#1  0x00007f7c050a3b76 in ?? () from /usr/lib64/libxenstore.so.3.0
#2  0x00007f7c050a3c2f in ?? () from /usr/lib64/libxenstore.so.3.0
#3  0x00007f7c050a3f72 in ?? () from /usr/lib64/libxenstore.so.3.0
#4  0x00007f7c055f10a4 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c03c9104d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f7bfe4b0700 (LWP 7288)):
#0  0x00007f7c055f505f in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x00007f7c07199d99 in ?? ()
#2  0x00007f7c0709a213 in ?? ()
#3  0x00007f7c0709a610 in ?? ()
#4  0x00007f7c055f10a4 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c03c9104d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f7c07483980 (LWP 7286)):
#0  0x00007f7c03c88cdf in ppoll () from /lib64/libc.so.6
#1  0x00007f7c07060ec9 in ?? ()
#2  0x00007f7c07026654 in ?? ()
#3  0x00007f7c06edcb36 in ?? ()
#4  0x00007f7c03bcdb05 in __libc_start_main () from /lib64/libc.so.6
#5  0x00007f7c06ee0eec in ?? ()

My question is how to resolve the function name from the address? such as 0x00007fe4ab73eb36. I know maybe through installing debug-info packages, but how to know install which packages?

Update:
According to Mark Plotnick's comments, I use the following command to get which debuginfo packages are lacked: 

linux:~ # gdb --quiet -nx --readnever /proc/7286/exe 7286
......
Missing separate debuginfos, use: zypper install glibc-debuginfo-2.19-31.5.x86_64 ......

After installing all need debuginfo packages, the symbols can be resolved: 

linux:~ # pstack 7286
Thread 3 (Thread 0x7f7c074b5700 (LWP 7287)):
#0  0x00007f7c055f7a9d in read () from /lib64/libpthread.so.0
#1  0x00007f7c050a3b76 in read_all.part.1.constprop ()
#2  0x00007f7c050a3c2f in read_message.constprop ()
#3  0x00007f7c050a3f72 in read_thread () from /usr/lib64/libxenstore.so.3.0
#4  0x00007f7c055f10a4 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c03c9104d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f7bfe4b0700 (LWP 7288)):
#0  0x00007f7c055f505f in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x00007f7c07199d99 in qemu_cond_wait ()
#2  0x00007f7c0709a213 in vnc_worker_thread_loop ()
#3  0x00007f7c0709a610 in vnc_worker_thread ()
#4  0x00007f7c055f10a4 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c03c9104d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f7c07483980 (LWP 7286)):
#0  0x00007f7c03c88cdf in ppoll () from /lib64/libc.so.6
#1  0x00007f7c07060ec9 in qemu_poll_ns ()
#2  0x00007f7c07026654 in main_loop_wait ()
#3  0x00007f7c06edcb36 in main ()

But "objdump -t /proc/7286/exe | grep main" outputs nothing: 

linux:~ # objdump -t /proc/7286/exe | grep main
linux:~ #
Nan Xiao
  • 16,671
  • 18
  • 103
  • 164
  • Do you finding a programming way to resolve the address or a tool ? – Zang MingJie Oct 22 '15 at 06:40
  • @ZangMingJie: A out-of-box tool is best! – Nan Xiao Oct 22 '15 at 07:05
  • 2 days no answer means no such tool – Zang MingJie Oct 24 '15 at 12:26
  • 1
    If you don't include that `| /bin/sed` bit, does gdb list any specific executables or shared libraries accompanied by a "no debugging symbols found" warning? – Mark Plotnick Oct 27 '15 at 17:55
  • @MarkPlotnick: Yes, you are right! `gdb --quiet -nx --readnever /proc/$pid/exe $pid` can list which debug-info packages should be installed, You can also post the answer, and I can accept it. thx! – Nan Xiao Oct 28 '15 at 10:17
  • Looking more closely at your stack trace, I think the main problem may be that your executable might be stripped, i.e. no symbol table, since there's not even a frame for `main` above `__libc_start_main`. Can you run `objdump -t /proc/$pid/exe | grep main` and see if there's a symbol table entry for it? – Mark Plotnick Oct 28 '15 at 15:16
  • @MarkPlotnick: Yes, "`objdump -t /proc/$pid/exe | grep main`" shows nothing. I have updated the OP. – Nan Xiao Oct 29 '15 at 02:52
  • I don't know why installing glibc debugging symbols would cause symbols in executable files (aside from the standard shared libraries) to suddenly appear. Is your executable an ordinary x86_64 C program? I see `qemu_cond_wait`; are you running inside `qemu`? – Mark Plotnick Oct 29 '15 at 17:16
  • @MarkPlotnick: the process `7286` is `qemu-system-i386`, which shipped by `SuSE`. – Nan Xiao Oct 30 '15 at 00:08

0 Answers0