2
gdb -quiet -iex 'set pagination off' -ex run -ex 'thread apply all bt' --batch --args <your prog>

The above is my default way or running my programs in CI. It is very convenient to have a stacktrace printed if the binary crashes, without having to hunt for coredump files.

edit: my default way of running CI has changed, because I also need to propagate the return code from the tested program in some circumstances

gdb -quiet -iex 'set pagination off' -iex 'set debuginfod enabled on' -ex run -ex 'thread apply all bt' -ex 'quit $_exitcode' --batch --args <your prog>

The problem is that I cannot do this when I compile with -fsanitize=address,leak,undefined. I get an error message when the program runs to the end and lsan is triggered (in an atexit handler, according to its docs).

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
==2861213==LeakSanitizer has encountered a fatal error.
==2861213==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==2861213==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
[Inferior 1 (process 2861213) exited with code 01]

Q1: Can I maintain the convenience of my gdb -quiet setup, while still getting lsan leak reports when my program leaks?

Currently the solution I am implementing is to hunt for the coredump files and execute gdb on them in a subsequent CI step (that runs upon test failure). For this, I had to configure sanitizers to permit coredump generation with disable_coredump=0, as described in How to generate core dump when use Address Sanitizer and gcc7.1.0.

Q2: Is it possible to use AddressSanitizer to do the job gdb used to do for me? That is, to run the equivalent of bt or thread apply all bt when my program crashes?

user7610
  • 25,267
  • 15
  • 124
  • 150
  • 1
    `ASAN_OPTIONS="handle_abort=1"` appears to print the backtrace of the thread that hit the `abort()`. If I set that, there does not seem to be a way to get both that stacktrace and a coredump file (`disable_coredump=0` then has no effect). See https://stackoverflow.com/a/42866567/1047788. – user7610 Mar 05 '22 at 22:49
  • `handle_abort=1` can sometimes fail without printing any stacktrace, only prints `AddressSanitizer: nested bug in the same thread, aborting.` Dumping core and running gdb in this situation does provide a stacktrace for me. – user7610 Mar 06 '22 at 09:33

1 Answers1

1
export DEBUGINFOD_URLS=https://debuginfod.elfutils.org
gdb -quiet -iex 'set pagination off' -iex 'set debuginfod enabled on' -iex 'set detach-on-fork off' -iex 'set breakpoint pending on' -x gha_gdb_commandfile.txt --args <your prog> <args>

The gdb_commandfile.txt being

break __lsan::CheckForLeaks
commands
detach
end

run
thread apply all bt
thread apply all py-bt

The gdb command file breaks when leak sanitizer is starting and detaches the debugger, so that leak sanitizer is free to do its own attach.

I tried this to debug this main.c

#include <assert.h>
#include <stdlib.h>

int main(int argc, char** argv) {
        void* x = malloc(42);
        if (argc > 1) assert(1 == 2);
        return 0;
}

compiled as

gcc -g3 -fsanitize=address,leak main.c

and it gives both leak report and backtrace, depending on whether I run with argument or not.

The command works even if binary is compiled without leak sanitizer. Then gdb simply never sets the breakpoint.

I wanted to have gdb watch both parent and child if my process starts a subprocess. I tried to use 'set detach-on-fork off' for it. I need it because there might be subprocesses that should be leak-checked (and stacktrace should be dumped if they crash). But what I tried is not the way. The parent process should instead start the child within new instance of GDB. By default, GDB continues attached to parent, ignoring the child.

The rr debugger is able to record entire process subtree. Maybe there can be a way. But that is for another question.

user7610
  • 25,267
  • 15
  • 124
  • 150