No backtrace from SIGABRT signal on ARM platform?

Question

I'm using 'backtrce()' and 'backtrace_symbols_fd()' functions in a signal handler to generate a backtrace for debugging (GDB not available).

They work fine on x86 desktop (Ubuntu), but on the target device (ARM based) the backtrace on Abort signal (due to double-free error) shows only three frames: the signal handler and two from within libc, which is not useful for debugging our code! Backtrace on SEGV (e.g. using a bad pointer) DOES produce a good backtrace.

Why can't I get a useful backtrace on ABRT signal on ARM?

[Question edited for clarity]

Here's a simple test program which demonstrates the problem:

#include <execinfo.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

// Signal hangler to catch seg fault:
void handler_segv(int sig) {
    // get void*'s for all entries on the stack
    void *array[10];
    size_t size;
    size = backtrace(array, 10);
    fprintf(stderr, "Error: Signal %d; %d frames found:\n", sig, size);
    // print out all the frames to stderr
    backtrace_symbols_fd(array, size, STDERR_FILENO);
    exit(1);
}


void crashme()
{
  // Deliberate Error: Abort (double free):
  char *test_ptr = malloc(1);
  free(test_ptr);
  free(test_ptr);
  // Deliberate Error #2: Seg fault:
  //char * p = NULL;
  //*p = 0;
}

void foo()
{
    fprintf(stdout, "---->About to crash...\n");
    crashme();
    fprintf(stdout, "---->Crashed (shouldn't get to here)...\n");
}



// Main entry point:
int main(int argc, char *argv[])
{
    fprintf(stdout, "Application start...\n");

    // Install signal handlers:
    fprintf(stdout, "-->Adding handler for SIGSEGV and SIGABRT\n");
    signal(SIGSEGV, handler_segv);
    signal(SIGABRT, handler_segv);

    fprintf(stdout, "-->OK. Causing Error...\n");
    foo();
    fprintf(stdout, "-->Test finished (shouldn't get to here!)\n");
    return 0;
}

This was compiled for x86 as follows:

gcc -o test test-backtrace-simple.c -g -rdynamic

And for ARM:

arm-none-linux-gnueabi-gcc -o test-arm test-backtrace-simple.c -g -rdynamic -O0 -mapcs-frame -funwind-tables -fasynchronous-unwind-tables

I've used various compiler options for ARM as described in other posts related to generating backtraces on ARM.

When run on the x86 desktop, it generates the expected output with plenty of debug, ending in:

Error: Signal 6; 10 frames found: 
./test(handler_segv+0x19)[0x80487dd]
[0xb7745404] 
[0xb7745428]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb75b0e0f]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb75b4455]
/lib/i386-linux-gnu/libc.so.6(+0x6a43a)[0xb75ed43a]
/lib/i386-linux-gnu/libc.so.6(+0x74f82)[0xb75f7f82]
./test(crashme+0x2b)[0x8048855] 
./test(foo+0x33)[0x804888a]
./test(main+0xae)[0x8048962]

(i.e. the back trace generated by my handler, with my function calls at the bottom).

However, when run on the ARM platform, I get:

Application start...
-->Adding handler for SIGSEGV and SIGABRT
-->OK. Causing Error...
---->About to crash...
*** Error in `/opt/bin/test-arm': double free or corruption (fasttop): 0x015b6008 ***
Error: Signal 6; 3 frames found:
/opt/bin/test-arm(handler_segv+0x24)[0x8868]
/lib/libc.so.6(__default_sa_restorer_v2+0x0)[0xb6e6c150]
/lib/libc.so.6(gsignal+0x34)[0xb6e6af48]

The backtrace() finds only 3 frames, and they are only the signal handler and something in libc (not useful)!

I found a mailing list post which said:

If you link with the debugging C library, -lc_g, you'll get debugging info back past abort().

This might be relevant, but -lc_g doesn't work on my compiler (ld: cannot find -lg_c).

The backtrace works fine on ARM if I generate a seg fault instead (e.g. change crashme() function to use "char *p = NULL; *p = 0;" instead of the double free.

Any ideas or suggestions for other ways to get a back trace?

[--EDIT--]

I tried some MALLOC_CHECK_ options as suggested in the comments, but the only effect was to change whether the abort was generated. Here is the output from three runs on the ARM:

 # MALLOC_CHECK_=0 /opt/bin/test-arm
Application start...
-->Adding handler for SIGSEGV and SIGABRT
-->OK. Causing Error...
---->About to crash...
---->Crashed (shouldn't get to here)...
-->Test finished (shouldn't get to here!)


# MALLOC_CHECK_=1 /opt/bin/test-arm
Application start...
-->Adding handler for SIGSEGV and SIGABRT
-->OK. Causing Error...
---->About to crash...
*** Error in `/opt/bin/test-arm': free(): invalid pointer: 0x015b2008 ***
---->Crashed (shouldn't get to here)...
-->Test finished (shouldn't get to here!)


# MALLOC_CHECK_=2 /opt/bin/test-arm
Application start...
-->Adding handler for SIGSEGV and SIGABRT
-->OK. Causing Error...
---->About to crash...
Error: Signal 6; 3 frames found:
/opt/bin/test-arm(handler_segv+0x24)[0x8868]
/lib/libc.so.6(__default_sa_restorer_v2+0x0)[0xb6e24150]
/lib/libc.so.6(gsignal+0x34)[0xb6e22f48]
#

MALLOC_CHECK_=0: No error message (double free is ignored!)

MALLOC_CHECK_=1: Error message, but program continues

MALLOC_CHECK_=2: Error message and ABRT signal; useless backtrace generated (this is the default behaviour!)

My cross compiler reports: gcc version 4.6.1 (Sourcery CodeBench Lite 2011.09-70) Target device has linux kernel version 3.8.8

Answer 1

It appears you have done sufficient research to know that you need the switches -funwind-tables and -fasynchronous-unwind-tables in your compiler command line. In practice either one of them seems sufficient but clearly without them backtracing doesn't work at all. Now, the trouble with things like SIGABRT is that the backtrace must traverse stack frames that were generated by libc functions such as abort and gsignal, and fails because that lib is not built with either of those switches (in any distribution that I know of).

While it would be nice to petition the maintainers of Sourcery CodeBench to build their distribution with that option, the only immediate solution is to build libc yourself, with either or both of those flags set (in my experience just -funwind-tables is enough). If you also need a stack trace in case of catching an unhandled exception (via std::set_terminate) then you will also need to rebuild libstdc++.

At my workplace we needed backtraces for both cases (SIGABRT and unhandled exceptions), and since libstdc++ is part of the toolchain we rebuilt the toolchain ourselves. The tool crosstool-NG makes this relatively easy to do. In the configuration utility ./ct-ng menuconfig we entered section Target Options and edited Target CFLAGS (which sets the build variable TARGET_CFLAGS) to -funwind-tables. The resulting toolchain (more specifically, using the libc and libstdc++ from the resulting toolchain build) provides us with a full backtrace in nearly all cases.

I've found one case where we still don't get a full backtrace: if the crash occurred within a function that originally is written in assembly, such as memcpy (unfortunately this is not an uncommon occurrence). Perhaps some option needs to be passed to the assembler, but I didn't have the time to investigate this further.

Answer 2

This is because unwinding through signal handlers is broken in glibc on ARM. I've dug into this a few years back and managed to create a working standalone fix. The hard part was digging through the undocumented bowels of exception handling in glibc, after that the fix was simple bordering on trivial.

I posted this to the glibc mailing list, as reply to an old thread about this problem, in the hope that a glibc dev would take my standalone fix as guide to fix it in glibc proper, but this never happened.

Recently I tested it again: it turns out that the problem still hasn't been fixed in glibc, and due to changes in glibc my fix no longer works. Update: I've fixed it!