11

I'm developing on Android 2.3.x using NDK r5b. Occasionally my code crashes and I'd like to know where. I already know how to get the corresponding line in my application when I have a have a pointer (i.e. from Android's stack traces.)

However, oftentimes I see useless stack traces like this (full stack trace):

     #00  pc 0006561a  /system/lib/egl/libGLESv2_adreno200.so
     #01  pc 0006b900  /system/lib/egl/libGLESv2_adreno200.so
     #02  pc 0005aac8  /system/lib/egl/libGLESv2_adreno200.so
     #03  pc 0001687a  /system/lib/egl/libGLESv1_CM_adreno200.so
     #04  pc 000096ce  /system/lib/egl/libGLESv1_CM_adreno200.so

or this:

(gdb) bt
#0  0xafd0c51c in epoll_wait () from /Volumes/SecureCode/webos/rta/android/obj/local/armeabi/libc.so
#1  0xa81216a6 in ?? ()

that don't even mention my code at all.

Is there any way at all to get better stack traces than this? Why are some library functions "opaque" in that they don't allow the backtrace to "see through" to the calling function, causing a stop in the stack trace?

As far as I can tell, the only way to debug a problem like this is to use logging at each point in the program and/or step through each line with gdb.

Are there ROMs available with debug versions of these Android libraries instead of runtime ones, and would that help? (I use one phone solely for development, so I'm not concerned about keeping full functionality.) (Actually, I noticed that the path to libc.so in the above gdb stack trace is within my application directory. Could I possibly just package it with a different (debug) libc.so, and would that help?)

One last thing that might help: in the above stack trace from logcat (the first one,) my library is mentioned in the raw stack dump:

stack:
  ...
  ...
  4471cb88  00000028  
  4471cb8c  afd4649c  
  4471cb90  80b4eb71  /data/data/com.audia.dev.rta/lib/librta.so
  4471cb94  00299180  
  ...
  ...

but that is not a function pointer. What could that be, and would it be of any help after the app has crashed? I'm guessing probably not if it's a heap pointer or something like that.

tmandry
  • 1,295
  • 16
  • 33

5 Answers5

6

Is there any way at all to get better stack traces than this?

As far as I know, you must build and write the Android image by yourself. It enables you to have the whole complete symbols of the Android (executable files and shared libraries) except proprietary shared libraries.

Also it provides to use the symbols using gdb.

$ adb shell setprop debug.db.uid 32767
$ adb forward tcp:5039 tcp:5039

/*
 program terminated and debuggerd caught exception like the following.
 Use the PID number for gdbclient 3rd parameter.
 I/DEBUG   ( 2154): ******************************************************** 
 I/DEBUG   ( 2154): * Process 2508 has been suspended while crashing.  To
 I/DEBUG   ( 2154): * attach gdbserver for a gdb connection on port 5039:
 I/DEBUG   ( 2154): *
 I/DEBUG   ( 2154): *     adb shell gdbserver :5039 --attach 2508 &
 I/DEBUG   ( 2154): *
 I/DEBUG   ( 2154): * Press HOME key to let the process continue crashing.
 I/DEBUG   ( 2154): ********************************************************)
*/

$ gdbclient "" "" 2508

EDITED:

You can still use ndk-gdb instead of gdbclient command. Please specify the symbol files for shared libraries.

(gdb) set solib-search-path (ANDROID_SOURCE_PATH)/out/target/product/(PRODUCT_NAME)/symbols/system/lib

EDITED 2:

If you don't need the symbols of the Android system shared libraries, just adb pull shared libraries and set sollib-search-path to it.

$ adb pull /system/lib lib

$ ndk-gdb
...
(gdb) set solib-search-path lib
Kazuki Sakamoto
  • 13,929
  • 2
  • 34
  • 96
  • Sounds promising. What are the significance of the numbers 32767 and 2508? – tmandry Jun 27 '11 at 07:24
  • "setprop debug.db.uid 32767" means debuggerd waits for user action when the process uid is lower equal than the specified number(UID). 'gdbclient "" "" 2508' means gdbserver attaches to the process that the process ID is 2508(PID) as debuggerd's message. – Kazuki Sakamoto Jun 27 '11 at 07:31
  • Thanks. So most of this can be done without compiling Android yourself, but the one benefit is that you can compile Android libs with debugging turned on and get symbols? – tmandry Jun 27 '11 at 17:06
  • Updated my answer for just tracing stack without symbols. – Kazuki Sakamoto Jun 28 '11 at 02:20
  • Okay, that makes sense. I can see this as being very helpful. Note: in either use case you should also add the directory that contains your project libs to `solib-search-path` (i.e. under `obj/local/armeabi`), which is what happens automatically when using `ndk-gdb`. – tmandry Jun 28 '11 at 04:03
  • Oddly enough, running `setprop debug.db.uid 32767` causes the stack trace to not be printed in the log, and no debuggerd messages. The process is terminated immediately instead (one Zygote log line confirms this) so there is nothing to attach to. This is the behavior regardless of the value I set (I can't undo it.) Is this because I'm using a stock ROM? (I have root but no custom ROM.) – tmandry Jun 29 '11 at 06:09
  • I heard that debuggerd doesn't work on some devices. What device are you using? Also please refer to [android-platform ML's mail](https://groups.google.com/group/android-platform/browse_thread/thread/fee9ca74d48fa64e). – Kazuki Sakamoto Jun 29 '11 at 06:20
  • Hmm, at least, debuggerd of Nexus One works fine for me... Also [Compile CyanogenMod for Passion](http://wiki.cyanogenmod.com/wiki/Compile_CyanogenMod_for_Passion) is quite helpful. – Kazuki Sakamoto Jun 29 '11 at 07:34
4

Couple of notes:

  • In some cases, your stack trace may be damaged because your stack has been partially trashed. Unlikely though.
  • What OS are you using? Gingerbread (Android 2.3) is much better in terms of stack traces. If you're not running Android 2.3, find an Android 2.3 ROM for your phone somewhere, or get a cheap development phone that runs 2.3.
  • Have you seen Onur's script? It's been working very well for me, even on Android 2.2 phones.
  • Hope that fadden is reading this, I'm sure he has an answer that is much more helpful than mine.
EboMike
  • 76,846
  • 14
  • 164
  • 167
  • I am running Gingerbread, yes. I'll look at the script, but can it do anything more than opening up the library with `gdb` and running `list *0xabcd1234` can? (It may be more convenient, yes, but what I'm really looking for is a way to get more information that I can currently.) – tmandry Jun 15 '11 at 06:44
  • Out of curiosity: did you ever try the above suggestion? I use a similar method (less scripty, but still using addr2line) to get function name, module name, and line numbers from those stack traces. – jimkberry Jun 23 '11 at 20:47
  • 2
    Oh, and something that was not obvious to me initially: If you are using the NDK build scripts, don't run addr2line against .so files in lib/armeabi, since they are stripped by the "install' command that puts them there and so can't tell you anything. Use the files in obj/local/armeabi instead. – jimkberry Jun 23 '11 at 21:00
  • The problem with addr2line is that it requires an address, and some stack traces (like the one above) just don't give an address in your code. For the stack traces that do, I generally have no problem getting what I need. – tmandry Jun 24 '11 at 03:38
1

Check out the following question: How to generate a stacktrace when my gcc C++ app crashes

We did the same with our Android app.: we wrote our own signal handler, handled signals 7 (sigbus) and 11 (sigsegv) and printed out the stack tace from the handler. We didn't use the backtrace() function though, but unwinded the stack manually...

Combining the first two answers you should be able to write your own signal handler to dump the stack trace. This article may help you also: http://www.ibm.com/developerworks/power/library/l-sigdebug/index.html. Only keep in mind, that extracting register content is architecture dependent, so you must substitute the structures used in the above codes with the ones on android (ARM processor dependent). For example I had to dig into the Android source for 'struct ucontext'.

When you have the stack trace, run a script on the output, which will resolve the symbols using addr2line and your unstripped executable.

Community
  • 1
  • 1
bxantus
  • 41
  • 3
  • This is helpful in retrieving and sending a stack trace when the app is in production, but does it actually provide a more informative stack trace than the default Android one? – tmandry Jun 27 '11 at 17:03
  • Yes, I belive so. You get a full native stack trace of your App. The default Android stack trace contains the Java stack, and stops at the first entry in the native stack trace (at least we did experience that). The stack traces, you mentioned in your question, look like various system wide traces – bxantus Jun 28 '11 at 08:11
  • No, most stack traces go into the app itself, because the errors happen in the native side (this is where most of the app lives.) They are always caused by the app, but too often the stack trace will stop before it gets to my .so, hence the question. – tmandry Jun 28 '11 at 17:55
1

Sorry to flood my own question with answers, but I did discover that integrating Google Breakpad was an excellent way to get good stack traces/crash reports. It's easy to write a signal handler that calls Breakpad, and that takes care of everything; we just have to upload the reports to our server. We also integrated the process of calling stackwalk.sh into our build system. It took some work, but all in all it's perfect for getting good native crash reports on Android.

This answer has some details on writing a signal handler; the rest of the code you need is on the Google Breakpad site under the wiki.

Community
  • 1
  • 1
tmandry
  • 1,295
  • 16
  • 33
0

I did get an answer to this question for some stack traces. (From the looks of this question that may be all I get.) These are the ones that terminate with an lr (link register) address. See my other question/answer.

Community
  • 1
  • 1
tmandry
  • 1,295
  • 16
  • 33