How to investigate Xamarin/libmonosgen-2.0.so native crash?

Question

My latest release is crashing at a rate of about 99 per 1000 devices - significantly higher than the release before it. New crashes come in 2 flavors in libmonosgen:

Cluster 1: libmonosgen-2.0.so

signal 11 (SIGSEGV), code 2 (SEGV_ACCERR)
libmonosgen-2.0.so

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> com.dailybits.foodjournal <<<

backtrace:
  #00  pc 0000000000192304  /data/app/com.dailybits.foodjournal-KapbLS3Zx20G2S79yI3CTw==/lib/arm/libmonosgen-2.0.so

Cluster 2: libmonosgen-2.0.so (mono_class_get_flags)

signal 11 (SIGSEGV), code 2 (SEGV_ACCERR)
mono_class_get_flags
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> com.dailybits.foodjournal <<<

backtrace:
  #00  pc 000000000014f310  /data/app/com.dailybits.foodjournal-rwLqwfyotHI-Ulk481Wk8g==/lib/arm64/libmonosgen-2.0.so (mono_class_get_flags)

Pre-release report is looking all green, and App Center has none of these crashes, they only show up in Google Play.

Questions:

1. Does anyone know what pid:0 and tid:0 mean? Does this mean these crashes are occurring before the application code is loaded, or is that unrelated?

2. If this isn't at app startup, can I somehow find out after how long the crash occurs? Or even better, can I correlate the crashes to app generated session logs?

3. Is there a way to configure Play, or the App to collect better traces?

Configuration details below

(both build options appear to result in the same problem)

Version info

Microsoft Visual Studio Professional 2019
Version 16.4.2
VisualStudio.16.Release/16.4.2+29613.14
Microsoft .NET Framework
Version 4.8.03752

Xamarin   16.4.000.307 (d16-4@e031886)
Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin.Android SDK   10.1.1.0 (d16-4/f2c9364)
Xamarin.Android Reference Assemblies and MSBuild support.
    Mono: bef1e63
    Java.Interop: xamarin/java.interop/d16-4@c4e569f
    ProGuard: xamarin/proguard/master@905836d
    SQLite: xamarin/sqlite/3.28.0@46204c4
    Xamarin.Android Tools: xamarin/xamarin-android-tools/master@9f4ed4b

Edit 2/17/2020

Some of my investigation to date:

1. Looks like libmonosgen contains the Xamarin garbage collector.
2. SEGV_ACCERR is an access denied on a platform object (IE object already disposed). A good way to get these apparently is to have static objects accessed from multiple threads. I'm thinking this may mean one of the threads disposes the underlaying Java object, without telling the Mono runtime.
3. Third party libraries, as well as new Java objects (IE changes to UI controls I'm using) are suspects.

I've decided to go with a fine-toothed comb through all the changes, revert all the package updates, and re-do them one-by-one, until a release starts crashing. If I'm unlucky, it's going to take a few weeks, but it should work to find out what causes the crash. It should also allow me to ship my new features to 100% without creating a spike in crashes for all users. Fingers crossed :)

Answer 1

After a good amount of research (see bottom of question), I decided to strip all changes from the app, and start dishing them out again one by one. Fortunately, just as I was done doing a detailed comb-through of all changes (from 2 months of work), a user posted a 1 star review, with detailed repro instructions!.

So, the answer to the question "how to investigate crashes with no stack trace":

You have to find a repro, and one way (for on production releases) to find one is to wait for your users to tell you.

Answers to the other in-line questions:

1. Does anyone know what pid:0 and tid:0 mean? Does this mean these crashes are occurring before the application code is loaded, or is that unrelated?

   • This means the stack got corrupted, and process and tid are no longer available (does not mean that this is at startup). IE Weird Native Crash - pid: 0, tid: 0 signal 11 (SIGSEGV), code 1 (SEGV_MAPERR)

2. If this isn't at app startup, can I somehow find out after how long the crash occurs? Or even better, can I correlate the crashes to app generated session logs?

   • I haven't found a way to add to these logs, the better way is to find a repro.

3. Is there a way to configure Play, or the App to collect better traces?

   • Not for these kinds of crashes.

---

Details / root cause of the actual crash:

As it turns out, my favorite PropertyChanged.Fody and Xamarin Forms's behavior had changed since the last time I had updated it, to the point where more PropertyChanged events were generated than in the past (IE, even if the value that is being assigned is equal to the original value, a "propertychanged" event is now fired).

In one of my more interesting viewmodels (that does automatic unit conversion based on the text value from an entry control), and when combined with a custom control this results into an infinite recursion. (Meaning the value would change, which would trigger a propertychanged event, which would update the control, which would assign the value to the viewmodel, which would trigger propertychanged, etc...)

Once I had a repro, investigating and fixing the issue was trivial; I was able to solve the issue, by decorating one of the properties with [DoNotNotify]