0

We recently hit some issues that may be related to the GC behavior of CLR.

The problem I encountered is as follows:

We have a long running stress testing application written in C# that keeps opening file handles on a remote SMB file share (which is Azure Files Service), and uses those handles to perform file system operations like read/write, etc.

Typically we’ll keep those handle open for quite a long time, as we’ll use them repeatedly. But sometimes when we try to access some of those opened handles, we found that these handles were closed already. And from the trace logs captured by Process Monitor (one sample below):


fltmgr.sys!FltpPerformPreCallbacks+0x324
fltmgr.sys!FltpPassThroughInternal+0x8c
fltmgr.sys!FltpPassThrough+0x169
fltmgr.sys!FltpDispatch+0x9e
ntoskrnl.exeIopCloseFile+0x146
ntoskrnl.exeObpDecrementHandleCount+0x9a
ntoskrnl.exeNtClose+0x3d9
ntoskrnl.exeKiSystemServiceCopyEnd+0x13
ntdll.dll!ZwClose+0xa
KERNELBASE.dll!CloseHandle+0x17
mscorlib.ni.dll!mscorlib.ni.dll!+0x566038
clr.dll!CallDescrWorkerInternal+0x83
clr.dll!CallDescrWorkerWithHandler+0x4a
clr.dll!DispatchCallSimple+0x60
clr.dll!SafeHandle::RunReleaseMethod+0x69
clr.dll!SafeHandle::Release+0x152
clr.dll!SafeHandle::Dispose+0x5a
clr.dll!SafeHandle::DisposeNative+0x9b
mscorlib.ni.dll!mscorlib.ni.dll!+0x48d9d1
mscorlib.ni.dll!mscorlib.ni.dll!+0x504b83
clr.dll!FastCallFinalizeWorker+0x6
clr.dll!FastCallFinalize+0x55
clr.dll!MethodTable::CallFinalizer+0xac
clr.dll!WKS::CallFinalizer+0x61
clr.dll!WKS::DoOneFinalization+0x92
clr.dll!WKS::FinalizeAllObjects+0x8f
clr.dll!WKS::FinalizeAllObjects_Wrapper+0x18
clr.dll!ManagedThreadBase_DispatchInner+0x2d
clr.dll!ManagedThreadBase_DispatchMiddle+0x6c
clr.dll!ManagedThreadBase_DispatchOuter+0x75
clr.dll!ManagedThreadBase_DispatchInCorrectAD+0x15
clr.dll!Thread::DoADCallBack+0xff
clr.dll!ManagedThreadBase_DispatchInner+0x1d822c
clr.dll!WKS::DoOneFinalization+0x145
clr.dll!WKS::FinalizeAllObjects+0x8f
clr.dll!WKS::GCHeap::FinalizerThreadWorker+0xa1
clr.dll!ManagedThreadBase_DispatchInner+0x2d
clr.dll!ManagedThreadBase_DispatchMiddle+0x6c
clr.dll!ManagedThreadBase_DispatchOuter+0x75
clr.dll!WKS::GCHeap::FinalizerThreadStart+0xd7
clr.dll!Thread::intermediateThreadProc+0x7d
KERNEL32.dll!BaseThreadInitThunk+0x1a
ntdll.dll!RtlUserThreadStart+0x1d

It seems that the handles were closed in CLR GC Finalizer thread. However, our handles are opened in the following pattern which should not be GC’ed:

We use P/Invoke to open a file handle and obtain a SafeFileHandle and use that SafeFileHandle to construct a FileStream, and we’ll save the FileStream object in another object defined as follows:

public class ScteFileHandle
{
    /// <summary>
    /// local file handle
    /// </summary>
    [NonSerialized]
    public FileStream FileStreamHandle;

    /*
     * Some other fields
     */

}

P/Invoke we use:

    [DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Auto)]
    public static extern SafeFileHandle CreateFile(
       string lpFileName,
       Win32FileAccess dwDesiredAccess,
       Win32FileShare dwShareMode,
       IntPtr lpSecurityAttributes,
       Win32FileMode dwCreationDisposition,
       Win32FileAttributes dwFlagsAndAttributes,
       IntPtr hTemplateFile);

SafeFileHandle fileHandle = Win32FileIO.CreateFile(fullFilePath, win32FileAccess, win32FileShare, IntPtr.Zero, win32FileMode, win32FileAttr, IntPtr.Zero);

FileStream fileStream = new FileStream(fileHandle, fileAccess, Constants.XSMBFileSectorSize);

One thing we’re sure of is that during the whole lifetime of our stress testing application, we definitely keep a reference to the ScteFileHandle object, so it will never be cleaned up by GC. However, we do have observed the SafeHandle referenced within the ScteFileHandle ‘s FileStream got finalized in CLR GC thread, as pasted in above trace log.

So I’m wondering what caused the SafeFileHandle to be GC’ed and if there’s any approach to avoid this ? I’m not familiar with the CLR GC behavior but from my perspective, the SafeFileHandle is not supposed to be GC’ed.

Any pointer or insight is greatly appreciated ! Please let me know if any other detail you need to diagnostic this issue : )

Nickyoung
  • 1
  • SafeFileHandle can be collected as any other object, it's not special in this regard, but it has a flag that specifies whether the object owns the handle it holds, and if not, it won't close the handle when the object is collected. It looks as though you're holding on to the handle though but can you please show your last method that is on the stack when it was collected? – Lasse V. Karlsen Aug 01 '18 at 11:00
  • "Having" a reference to an `ScteFileHandle` object doesn't protect it from GC. Having a reference that might be read in the future does. So if it's only a local variable, just having it in that variable (if you never access it again) won't save it. Or if it's in a field of your class, that's got a wider scope, but if provably nothing needs `this` again within that class, *it* can be collected even when one or more of its methods is still running. Famously, the GC is aggressive enough that it can collect objects even if their *constructor* is still running, if no other `this` access is required – Damien_The_Unbeliever Aug 01 '18 at 11:00
  • The GC is super aggressive, which means that unless you ensure you're holding on to the *objects* holding the handle, *they* might be collected even though you're not yet done with the handle, and thus start cleaning up things. – Lasse V. Karlsen Aug 01 '18 at 11:01
  • A typical problem with handles and such on objects is that in a method you grab the handle out of the field and into a local variable, then start doing P/Invoke. If GC kicks in and see that you have no more use for the actual object it might put it into the freachable queue for the finalizer to finalize before collecting it, and if this too manages to execute during your method, that finalizer might close the handle. Sort of like sawing off the branch you're sitting on. – Lasse V. Karlsen Aug 01 '18 at 11:02
  • 1
    The GC is so aggressive that it can even clean up the object on which you called the method, simply because the `this` reference is just a reference like any other at that point. – Lasse V. Karlsen Aug 01 '18 at 11:02
  • One way to ensure this is to use `GC.KeepAlive(yourObject);` at the end of your method that does P/Invoke, which is a non-optimizable/inlinable call that does nothing, except that because it explicitly cannot be optimized out ensures the object lives to tell the tale up until past that method call. – Lasse V. Karlsen Aug 01 '18 at 11:04
  • But without seeing the code which was executing at the time it is all conjecture. – Lasse V. Karlsen Aug 01 '18 at 11:04
  • Thanks @LasseVågsætherKarlsen for replying. Sorry for not describing my problem clearly. In our app, each handle is opened in a separate thread and will be used in another thread later. And after each handle is created, the ScteFileHandle which has reference to that SafeFileHandle will be stored in a static Dictionary variable which is alive during the lifetime of our app. Since we should be able to retrieve those handles from the static Dictionary anytime we want during the lifetime of the app, from my pespective, those handle should never got GC'ed ? – Nickyoung Aug 02 '18 at 03:56
  • Do you clone the SafeFileHandle or construct SafeFileHandle objects yourself? I ask because I seriously doubt there is a bug in the .NET runtime at this point regarding objects being collected with live references to them. As such I am inclined to believe that *you* believe you still have a valid reference to it but you don't, so it was collected. Again, there is nothing special in this regard with SafeFileHandle, it's an object like any other. – Lasse V. Karlsen Aug 02 '18 at 06:33
  • @LasseVågsætherKarlsen the SafeFileHandle is just obtained from P/Invoke like posted above:
    SafeFileHandle fileHandle = Win32FileIO.CreateFile(fullFilePath, win32FileAccess, win32FileShare, IntPtr.Zero, win32FileMode, win32FileAttr, IntPtr.Zero);     – Nickyoung Aug 02 '18 at 07:38
  • Yes, but I meant, do you clone this object, or construct new ones based on it? – Lasse V. Karlsen Aug 02 '18 at 07:48
  • @LasseVågsætherKarlsen No. Except for using the SafeFileHandle to construct FileStream, I've never cloned or constructed new ones based on it. – Nickyoung Aug 02 '18 at 08:43

0 Answers0