Crash in GC finalizer thread, what's the problem with "DestroyScout"?

Question

I'm facing with a .Net server application, which crashes on an almost weekly basis on a problem in a "GC Finalizer Thread", more exactly at line 798 of "mscorlib.dll ...~DestroyScout()", according to Visual Studio.

Visual Studio also tries to open the file "DynamicILGenerator.gs". I don't have this file, but I've found a version of that file, where line 798 indeed is inside the destructor or the DestroyScout (whatever this might mean).

I have the following information in my Visual Studio environment:

Threads :

Not Flagged >   5892    0   Worker Thread   GC Finalizer Thread mscorlib.dll!System.Reflection.Emit.DynamicResolver.DestroyScout.~DestroyScout

Call stack:

    [Managed to Native Transition]  
>   mscorlib.dll!System.Reflection.Emit.DynamicResolver.DestroyScout.~DestroyScout() Line 798   C#
[Native to Managed Transition]  
kernel32.dll!@BaseThreadInitThunk@12()  Unknown
ntdll.dll!__RtlUserThreadStart()    Unknown
ntdll.dll!__RtlUserThreadStart@8()  Unknown

Locals (no way to be sure if that $exception object is correct):

+       $exception  {"Exception of type 'System.ExecutionEngineException' was thrown."} System.ExecutionEngineException
    this    Cannot obtain value of the local variable or argument because it is not available at this instruction pointer,
            possibly because it has been optimized away.    System.Reflection.Emit.DynamicResolver.DestroyScout
    Stack objects   No CLR objects were found in the stack memory range of the current frame.   

Source code of "DynamicILGenerator.cs", mentioning the DestroyScout class (line 798 is mentioned in comment):

    private class DestroyScout
    {
        internal RuntimeMethodHandleInternal m_methodHandle;

        [System.Security.SecuritySafeCritical]  // auto-generated
        ~DestroyScout()
        {
            if (m_methodHandle.IsNullHandle())
                return;

            // It is not safe to destroy the method if the managed resolver is alive.
            if (RuntimeMethodHandle.GetResolver(m_methodHandle) != null)
            {
                if (!Environment.HasShutdownStarted &&
                    !AppDomain.CurrentDomain.IsFinalizingForUnload())
                {
                    // Somebody might have been holding a reference on us via weak handle.
                    // We will keep trying. It will be hopefully released eventually.
                    GC.ReRegisterForFinalize(this);
                }
                return;
            }

            RuntimeMethodHandle.Destroy(m_methodHandle); // <===== line 798
        }
    }

Watch window (m_methodHandle):

m_methodHandle  Cannot obtain value of the local variable or argument because 
                it is not available at this instruction pointer,
                possibly because it has been optimized away.
                System.RuntimeMethodHandleInternal

General dump module information:

Dump Summary
------------
Dump File:  Application_Server2.0.exe.5296.dmp : C:\Temp_Folder\Application_Server2.0.exe.5296.dmp
Last Write Time:    14/06/2022 19:08:30
Process Name:   Application_Server2.0.exe : C:\Runtime\Application_Server2.0.exe
Process Architecture:   x86
Exception Code: 0xC0000005
Exception Information:  The thread tried to read from or write to a virtual address
                        for which it does not have the appropriate access.
Heap Information:   Present

System Information
------------------
OS Version: 10.0.14393
CLR Version(s): 4.7.3920.0

Modules
-------
Module Name                                           Module Path   Module Version
-----------                                           -----------   --------------
...
clr.dll     C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll       4.7.3920.0
...

Be aware: the dump arrived on a Windows-Server 2016 computer, I'm investigating the dump on my Windows-10 environment (don't be mistaking on OS Version in the dump summary)!

Edit

What might the destroyscout be trying to destroy? That might be very interesting.

Answer 1

I don't know what exactly is causing this crash, but I can tell you what DestroyScout does.

It's related to creating dynamic methods. The class DynamicResolver needs to clean up related unmanaged memory, which is not tracked by GC. But it cannot be cleaned up until there are definitely no references to the method anymore.

However, because malicious (or outright weird) code can use a long WeakReference which can survive a GC, and therefore resurrect the reference to the dynamic method after its finalizer has run. Hence DestroyScout comes along with its strange GC.ReRegisterForFinalize code in order to ensure that it's the last reference to be destroyed.

It's explained in a comment in the source code

// We can destroy the unmanaged part of dynamic method only after the managed part is definitely gone and thus
// nobody can call the dynamic method anymore. A call to finalizer alone does not guarantee that the managed 
// part is gone. A malicious code can keep a reference to DynamicMethod in long weak reference that survives finalization,
// or we can be running during shutdown where everything is finalized.
//
// The unmanaged resolver keeps a reference to the managed resolver in long weak handle. If the long weak handle 
// is null, we can be sure that the managed part of the dynamic method is definitely gone and that it is safe to 
// destroy the unmanaged part. (Note that the managed finalizer has to be on the same object that the long weak handle 
// points to in order for this to work.) Unfortunately, we can not perform the above check when out finalizer 
// is called - the long weak handle won't be cleared yet. Instead, we create a helper scout object that will attempt 
// to do the destruction after next GC.

---

As to your crash, this is happening in internal code, and is causing an ExecutionEngineException. This most likely happens when there is memory corruption, when memory is used in a way it wasn't supposed to be.

Memory corruption can happen for a number of reasons. In order of likelihood:

• Incorrect use of PInvoke to native Win32 functions (DllImport and asscociated marshalling).
• Incorrect use of unsafe (including library classes such as Unsafe and Buffer which do the same thing).
• Multi-threaded race conditions on objects which the Runtime does not expect to be used multi-threaded. This can cause such problems as torn reads and memory-barrier violations.
• A bug in .NET itself. This can be the easiest to exclude: just upgrade to the latest build.

Consider submitting the crash report to Microsoft for investigation.

Edit from the author:
In order to submit a crash report to Microsoft, the following URL can be used: https://www.microsoft.com/en-us/unifiedsupport. Take into account that this is a paying service and that you might need to deliver your entire source code Microsoft in order to get a full analysis of your crash dump.