30

The magic incantation

LD_PRELOAD=/lib/libSegFault.so someapp

runs someapp with libSegFault.so providing backtrace information on a SIGSEGV as described in many different places.

Other than using signal(7)-like approaches to cause SIGABRT to invoke the SIGSEGV handler, is there some way to get libSegFault to provide backtrace information for assert(3) failures?

Rhys Ulerich
  • 1,242
  • 1
  • 12
  • 28

2 Answers2

41
env SEGFAULT_SIGNALS="abrt segv" LD_PRELOAD=/lib/libSegFault.so someapp

Note that the actual path to the preload library may differ. On my machine, I'd use

env SEGFAULT_SIGNALS="abrt segv" LD_PRELOAD=/lib/x86_64-linux-gnu/libSegFault.so some-64bit-app

or

env SEGFAULT_SIGNALS="abrt segv" LD_PRELOAD=/lib/i386-linux-gnu/libSegFault.so some-32bit-app

depending whether the application I was running was compiled 64-bit or 32-bit. (You can use file to check.)

The source tells us there are three environment variables that define how libSegFault.so behaves:

  • SEGFAULT_SIGNALS: The list of signals that cause a stack trace. The default is SIGSEGV. A defined but empty SEGFAULT_SIGNALS means no signals cause a stack trace. The supported values are segv, ill, abrt, fpe, bus on systems that have the SIGBUS signal, stkflt on systems that have the SIGSTKFLT signal, and all for all of these.

  • SEGFAULT_USE_ALTSTACK: If defined in the environment, libSegFault.so uses an altenate stack for the stack trace signals. This may come in handy if you are debugging stack corruption.

  • SEGFAULT_OUTPUT_NAME: If defined in the environment, the stack trace is written to this file instead of standard error.

To be honest, I found these initially by examining the library with strings /lib/libSegFault.so | sed -e '/[^0-9A-Z_]/ d'. All standard libraries (libSegFault.so having become a part of GNU C library) are tunable via environment variables, so using something like that command to dump any strings that look like environment variable names is a quick way to find stuff to search for. Doing a web search on "SEGFAULT_SIGNALS" "SEGFAULT_OUTPUT_NAME" produces a number of useful links; seeing that it was part of the GNU C library nowadays, I went to the source git archives, found the actual source file for the library, and posted my answer.

Nominal Animal
  • 38,216
  • 5
  • 59
  • 86
  • Thank you. Where is that behavior of SEGFAULT_SIGNALS documented? I spent some time digging around for details on libSegFault.so and kept coming up blank. – Rhys Ulerich Sep 23 '13 at 14:15
  • 1
    @RhysUlerich: At the source, https://sourceware.org/git/?p=glibc.git;a=blob;f=debug/segfault.c;hb=HEAD -- I'll add the descriptions gleaned from the sources to my answer, in case it helps somebody later on. – Nominal Animal Sep 23 '13 at 18:12
  • Also, sorry for not awarding the bounty earlier. Dunno why I thought it would happen once I accepted the answer. Again, thanks for the time you spent on this. – Rhys Ulerich Sep 24 '13 at 17:00
  • 1
    You don't need the fully qualified path name, `LD_PRELOAD=libSegFault.so` is fine if the library is in the dl path. – Fernando Silveira Sep 03 '14 at 15:57
  • @FernandoSilveira: I recommend not mentioning that to beginners, because they often end up with more than one version of their library installed in different library directions, causing grief and wails for help when nothing they do seems to have an effect. The fix, of course, is to verify that there is only one copy of the library in any of the standard library directories, but the locations vary from distribution to distribution (and even depending on whether you have a multilib installation or not). No, sir; it is definitely better to guide them to use full path names. – Nominal Animal Sep 04 '14 at 04:48
  • @NominalAnimal I still disagree (maybe even more): I don't see why someone would install two versions of `libSegFault.so` (as they come pre installed in 99% of the Linux distros, why and how would **a beginner** install two glibcs in a machine or even touch the `libSegFault.so` library?) and it is much more difficult for someone to find the right version of `libSegFault.so` than for LD to find the wrong version in the path. I'd never ask someone (specially a beginner) to call `/bin/ls` instead of `ls`. My 2 cents. – Fernando Silveira Sep 04 '14 at 10:18
  • @FernandoSilveira: Well, `libSegFault.so` is special, being a system library, and you do have a good argument there. However, I've answered here and elsewhere quite a few questions regarding how to interpose (standard) library functions, and the most common problem I've seen is mis-installing the (user's own) interposing library, and using only the library file name in `LD_PRELOAD`. It just seems to me that consistently using the full path is the easiest way to avoid the most common pitfall I've seen. I could be misguided or just plain wrong, though; so do feel free to disagree. – Nominal Animal Sep 04 '14 at 17:38
4

In a similar vein, the glibc exception handler writes a stack dump to /dev/console on heap corruption errors.

If you are running your executable in a non tty (i.e. a systemd process or other detached process), the crash output goes to /dev/null, which is not so useful.

There is an undocumented feature to redirect the output to /dev/stderr. Set the following environment variable:

export LIBC_FATAL_STDERR_=1

This can be used in conjunction with libSegFault.so for maximal forensics.

It is also worth mentioning that this might give you two stack traces if you also enable backtraces for SIGABRT, as the glibc first does a stack trace, then signals SIGABRT ... and then libSegFault gives a second stack trace.

Mark Lakata
  • 19,989
  • 5
  • 106
  • 123