Is there any way to tell GDB to wait for a process to start and attach to it?

Question

I have a process that is called by another process which is called by another process and so on ad nauseam. It's a child process in a long tool chain.

This process is crashing.

I would like to catch this process in GDB to understand why it's crashing. However, the only way I can think of is:

1. Start the original parent process in the commandline.
2. Poll ps -C <name process I want to catch> and get the PID.
3. Launch GDB, attached to that process's PID.

This is cumbersome but usually does the job. The problem is that the current failure runs very quickly, and by the time I capture the PID and launch GDB, it's already passed the failure point.

I would like to launch GDB and instead of:

(gdb) attach <pid>

I would like to do:

(gdb) attach <process name when it launches>

Is there any way to do this?

---

I am using GDB 7.1 on Linux.

Answer 1

Here is my script called gdbwait:

#!/bin/sh
progstr=$1
progpid=`pgrep -o $progstr`
while [ "$progpid" = "" ]; do
  progpid=`pgrep -o $progstr`
done
gdb -ex continue -p $progpid

Usage:

gdbwait my_program

Sure it can be written nicer but Bourne shell script syntax is painful for me so if it works then I leave it alone. :)

If the new process launches and dies too quickly, add 1 second delay in your own program for debugging.

Answer 2

On Mac OS X, you can use:

(gdb) attach --waitfor <process-name>

but this also is sometimes unable to capture processes that exit very quickly. I'm unsure if this is supported on any other platforms.

GDB Release Notes for Mac OS X v10.5 WWDC Seed

Answer 3

You can attach to a parent process and set follow-fork-mode child. This will make gdb debug child process instead of parent after forking. Also catch fork will be useful. This will make gdb stop after each fork. See docs.

Answer 4

Not exactly what you expect, but it might help you in debugging.

valgrind --trace-children=yes your_program

will check and print memory errors in all children of the process, with stack trace and some detail about the error (eg. in case of double-free, you'd get the stack trace of the first free).

Also, you might make the crashing process generate a core dump, and debug this post-mortem. See this answer for details.

Answer 5

One of the programs in Tom Tromey's gdb-helpers repository on GitHub is preattach, which will make gdb attach to the next process created with a given name. It requires the Linux systemtap program.

Answer 6

I've been facing a similar problem with something I'm trying to debug, and I came up with a solution using ldpreload, but after seeing Joeys answer I think I'll try that first. In case it's helpful to anyone else though here's the idea:

Create an LD_PRELOAD library to hook the exec* calls (there's plenty of guides on how to do this around, but if I do this I'll update my answer with the code), check the path used when passing through the exec* call, if it's our target then output a message with the PID to stderr and go into an infinite loop (with sleep to avoid massive CPU usage). Then you can attach with gdb and modify the register used in the loop to continue execution.

This may involve some inline ASM to make sure the compiler doesn't optimise the infinite loop in such a way that makes it hard to break out of. A more eloquent way of doing it would be to find a way to detect that gdb has attached then trigger a breakpoint ("asm("int3");" should do the trick on the latter).

Answer 7

If the process crashes or exits before you can attach to it, you can patch its entrypoint so it loops at startup. And then unpause it from gdb or from the shell. This script works for x86_64 and i386 binaries:

#!/bin/bash
#
# Patch an ELF binary's entry point to pause it at startup.
# And patch it when loaded to unpause it.

set -oe pipefail
type readelf objdump dd awk grep > /dev/null

syntax()
{
    echo "syntax: `basename $0` --test  elf-file"
    echo "        `basename $0` --patch elf-file"
    echo "        `basename $0` --unpause PID"
    echo "        `basename $0` --addr    PID"
    exit 1
}

if [ "$1" = "--unpause" -o "$1" = "-u" -o "$1" = "--addr" ]; then
    PID="$2"; [ -n "$PID" ] || syntax
    BASE=$(head -1 /proc/$PID/maps | cut -d- -f1)
    START=$(readelf -h "/proc/$2/exe" | grep Entry | \
        awk 'match($0, /: *0x(.*)/, m) { print m[1] }' )
    if [ "$1" = "--addr" ]; then
        echo -e "(gdb)\nset {char}(0x$BASE+0x$START+1) = 0x02"
    else
        echo -en '\x02' | dd of=/proc/$PID/mem bs=1 \
            seek=$((0x$BASE+0x$START+1)) conv=notrunc status=none
    fi
    exit
elif [ "$1" = "--test" ]; then
    TEST=1
elif [ "$1" != "--patch" ]; then
    syntax
fi

shift; ELF="$1"; [ -n "$ELF" ] || syntax

START=$(readelf -h "$ELF" | grep Entry | \
    awk 'match($0, /: *0x(.*)/, m) { print m[1] }' )

objdump -d "$ELF" | grep -m1 " ${START}:" || true
[ "$TEST" != "1" ] || { echo -e "\nNow patch the binary using --patch.\n"; exit 1; }

echo -en "\xEB\xFE" | dd of="$ELF" bs=1 \
    seek=$((0x$START)) conv=notrunc status=none

objdump -d "$ELF" | grep -m1 " ${START}:" || true

echo
echo "Program will pause at startup. Unpause it as root with:"
echo "    $0 --unpause \`pidof `basename "$ELF"`\`"
echo "Or find the address to restore at runtime from gdb:"
echo "    $0 --addr \`pidof `basename "$ELF"`\`"
echo