3

How can we solve an unhandled exception thrown by a finalizer, which clearly does not come from our code?

By event AppDomain.CurrentDomain.UnhandledException we once in a while log an Exception, which does not come from our code and is terminating the program. The stacktrace starts with the Finalize() method and is called on a class A, which we do not use anywhere.

Questions

  1. Can we somehow detect a library/NuGet/project, which is causing this?
  2. Can we maybe do some hard core hack like:
    • Changing GC behaviour (catching exceptions on finalizer, printing the faulted object ...)
    • Continously finding all instances of A in memory and either recognizing its origin (what created them or has reference on them) or calling their finalizer ourselves at the right moment in try catch?
  3. Something else?

THE SPECIFIC INFO:

The full stacktrace starts with frames System.ComponentModel.Component.Finalize() and System.IO.FileSystemWatcher.Dispose(Boolean disposing). The FileSystemWatcher is derived from the Component class - therefore the finalizer is called on the FileSystemWatcher.

We do not use class FileSystemWatcher anywhere in our code. It may come from some NuGet, but we use many of them. Our solution is wide, we have no clue, what could it cause. We use .Net Core 2.2 running in dockers on Linux.

The logged exception info:

 AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.); 
 InnerException: NullReferenceException: Object reference not set to an instance of an object. 

 Stacktrace: 
 at System.Threading.CancellationTokenSource.CallbackNode.<>c.<ExecuteCallback>b__10_0(Object s)
 at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
 --- End of stack trace from previous location where exception was thrown ---
 at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
 --- End of inner exception stack trace ---
 at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
 at System.IO.FileSystemWatcher.StopRaisingEvents()
 at System.IO.FileSystemWatcher.Dispose(Boolean disposing)
 at System.ComponentModel.Component.Finalize()
frakon
  • 1,948
  • 2
  • 22
  • 26
  • 1
    I would be interested in that lambda function. – Taekahn Oct 20 '19 at 16:17
  • 2
    The error is the `System.Threading.CancellationTokenSource.ExecuteCallbackHandlers`, called somewhere by the Finalize. The error is also coming from your application otherwise it would be a second chance exception. *While all this is handy to know, how is that going to solve your problem...* a) enhance the logging to be able to reproduce the problem, https://stackoverflow.com/questions/30326673/user-activity-logging-telemetry-and-variables-in-global-exception-handlers, b) open the solution in ILSply and save it as a C# Solution, Find FileWatcher in the NuGet package(s) causing the problem. – Jeremy Thompson Oct 20 '19 at 22:31
  • Is the stack trace from 1) the Finalize-rooted exception, or 2) from the inner exception? If it's from 1), can you dump the stack trace from the inner exception. Also how do you log the errors? Could that influence your findings ( nlog uses FileSystemWatcher for example) – Simon Mourier Oct 21 '19 at 08:07
  • @SimonMourier the stacktrace is combination of AggregateException stacktrace and inner NullReferenceException - so it is the full stacktrace available. We log errors by Serilog ... – frakon Oct 21 '19 at 09:28
  • 2
    @JeremyThompson Thank you very much for your suggestions, according to them we will try these two things: 1. b): Use ILSpy on whole solution 2. inspired by a) link: We will deploy on our testing environment the Debug version of the most failing microservice, then connect to it with the VS remote debugger and set it to break only on NullReferenceExceptions. When it breaks, we will investigate the callback (lambda function) object. Tomorrow we should have the answers. – frakon Oct 21 '19 at 09:39
  • May be you can try to break at the first chance exception and than have a look at the threads. – Daniel W. Oct 23 '19 at 14:33
  • 1
    It is just a [dumb bug](https://github.com/dotnet/corefx/issues/42079), update to the latest .netcore version to fix. Much could be said about this, but do note that Docker didn't help you much. Probably got you to lock-in the buggy version. In the age of agile it is important to keep updating. In general, a .NETCore version major.x.y is risky for low values of y, it didn't get exposed to enough real-life-usage yet. – Hans Passant Oct 24 '19 at 13:27
  • UPDATE: We DID NOT SOLVE it yet. 1. We tried ILSpy, it found FileSystemWatcher only in AspNetCore, but it failed to decompile some libs, so it may be also somewhere else. 2. We debugged remotely, broke at the exception, but it showed a window "no code is currently executing ..." and we could not investigate the exception context. – frakon Oct 25 '19 at 10:04
  • 1
    OUR NEXT TRIES WILL BE: upgrading to .Net Core 3, running as a console app without AspNetCore libs, checking older versions of the app to find out when this exception started, running on Windows instead of Linux, ... But we will do these steps in a month, since now we have to develop something else. If you have any new suggestions meanwhile, please drop them here. – frakon Oct 25 '19 at 10:04

1 Answers1

-1

In my opinion, the only solution remains here is to divide your program into some smaller subprograms if possible and run all of them individually to find the faulty plugin. The subprograms do not need to do a meaningful part of jobs that the whole program does. It is just for test.

Farhad Rahmanifard
  • 688
  • 3
  • 13
  • The finalizer is typically not being called as a a result of explicitly interacting with a 3rd party library; it gets called when the GC scans the heap of objects for references that could be freed. More info: https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/classes-and-structs/destructors – Per Lundberg Oct 26 '19 at 20:38
  • I suggest another way (changed my answer entirely)! – Farhad Rahmanifard Oct 27 '19 at 19:37
  • Better this way, I removed my downvote. However, I wonder if Hans Passant isnät on to something here: https://stackoverflow.com/questions/58444716/unhandled-exception-in-finalizer-not-from-our-code/58540783?noredirect=1#comment103406335_58444716 – Per Lundberg Oct 29 '19 at 07:14