6

Note: Symbols are showing up in crashlytics for our c++ library, the problem is that they aren't showing for system libraries like libc, libart, libbase, and libandroid_runtime.

We have some tricky crashes that happen entirely in the Android runtime and these are hard to debug without symbols. In firebase crashlytics we see the following stack trace:

Crashed: Thread :  SIGABRT  0x0000000000000000
#00 pc 0x4e574 libc.so 
#01 pc 0x4e540 libc.so 
#02 pc 0x5677d8 libart.so 
#03 pc 0x13ab0 libbase.so 
#04 pc 0x13090 libbase.so 
#05 pc 0x38cb6c libart.so 
#06 pc 0x39f7d8 libart.so 
#07 pc 0x1260e0 libandroid_runtime.so 
#08 pc 0x124ef4 libandroid_runtime.so 
#09 pc 0x124dc4 libandroid_runtime.so 
#10 pc 0x115468 libandroid_runtime.so

When I force a test crash in our C++ library by dereferencing a null pointer, I see the following backtrace in my local Android Studio console:

...snip...
      #06 pc 00000000002d7644  /apex/com.android.art/lib64/libart.so (art_quick_generic_jni_trampoline+148) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
      #07 pc 00000000002cdd64  /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+548) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
      #08 pc 00000000002f23d0  /apex/com.android.art/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+312) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
      #09 pc 00000000003839f4  /apex/com.android.art/lib64/libart.so (bool art::interpreter::DoCall<true, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+800) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
      #10 pc 00000000003813f4  /apex/com.android.art/lib64/libart.so (MterpInvokeVirtualRange+1368) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
      #11 pc 00000000002c8714  /apex/com.android.art/lib64/libart.so (mterp_op_invoke_virtual_range+20) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
...snip...

However, the same crash in crashlytics looks like this:

...snip...
#07 pc 0x222244 libart.so 
#08 pc 0x218964 libart.so 
#09 pc 0x284208 libart.so 
#10 pc 0x3e34ac libart.so 
#11 pc 0x800ba4 libart.so 
...snip...

How can we get crashlytics to include the information that is clearly in the crashdump?

Some notes on our build setup:

  • We have already followed https://firebase.google.com/docs/crashlytics/ndk-reports
  • Gradle Project 1 builds the native library, ensuring that debug is on and symbols are not stripped.
  • Gradle Project 2 builds the app and links in the library and tells crashlytics to upload native symbols
            firebaseCrashlytics {
                nativeSymbolUploadEnabled true
                unstrippedNativeLibsDir file("PATH/TO/UNSTRIPPED/DIRECTORY")
            }
Gerardo
  • 3,460
  • 1
  • 16
  • 18
Andrew Stromme
  • 2,120
  • 23
  • 30
  • 1
    I'd start with comparison of binaries you have locally and on target, just to be sure it's really not stripped. The second thing I'd do - see where gdb is trying to look for symbols when you open coredump from device and check if it's there. – c4pQ Jun 22 '22 at 11:36
  • @c4pQ Unfortuantely this is a rare crash and we haven't seen it on any of our devices, only on devices out in the wild, and so we don't have access to the coredump nor the binaries on the target. Thus far our only information has come from Firebase Crashlytics and Google Play Store Console. – Andrew Stromme Jun 22 '22 at 23:51
  • This question seems similar-ish https://stackoverflow.com/questions/54243889/native-crash-in-librart-so-all-on-android-8-0-missing-symbols, but also not resolved. – Andrew Stromme Jun 23 '22 at 00:00
  • But I bet you can install app to some test phone you have and see what is installed. Even better solution is to add a button causing crash (divide by 0) and test the whole error/stacktrace handling pipeline. What do you think? – c4pQ Jun 27 '22 at 08:20
  • 1
    @c4pQ Thank you! We tried this and were able to get normal stack traces when we triggered the crash manually. We did end up solving this specific issue because we got lucky and one of our test phones finally crash once with the same trace, which gave much more debugging info that allowed us to solve it. However, ideally we wouldn't need to rely on chance and could have this info in the uploaded stack trace. – Andrew Stromme Jun 30 '22 at 00:55

1 Answers1

1

This behavior is expected. When Crashlytics gets NDK crashes, these need to be symbolicated. In order to do this, the respective symbol files should be uploaded to Crashlytics.

With the configuration you mentioned, the symbols available in your app will be uploaded to Crashlytics.

nativeSymbolUploadEnabled true

But in the case of system libraries, the respective symbol files are not available (they are not public as far as I know). So Crashlytics won't have access to the required symbol files for symbolicating frames from system libraries.

Gerardo
  • 3,460
  • 1
  • 16
  • 18