0

Request assistance in analyzing an application crash issue. While pasting an image, I am using an external dll which is written in C++ code. Initially, we are registering a callback from C# code which after the paste operation, is invoking a callback function in C# code. After the callback function is finished, application is crashing without being caught in any try catch block. Crash dump analysis using WinDbg:

************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*
OK                                             D:\Core\Bin
Symbol search path is: srv*;D:\Core\Bin
Executable search path is: 
Windows 10 Version 19044 MP (8 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
Edition build lab: 19041.1.amd64fre.vb_release.191206-1406
Machine Name:
Debug session time: Wed Jul 13 15:55:25.000 2022 (UTC + 5:30)
System Uptime: 0 days 1:21:28.318
Process Uptime: 0 days 0:01:57.000
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
.....................
Loading unloaded module list
.........
For analysis of this file, run !analyze -v
eax=00000000 ebx=00c5e5e8 ecx=00000003 edx=00000000 esi=00000000 edi=000002f4
eip=778129fc esp=0019d0dc ebp=0019d14c iopl=0         nv up ei pl nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202
ntdll!NtWaitForSingleObject+0xc:
778129fc c20c00          ret     0Ch
0:000> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

*** WARNING: Unable to verify checksum for libcef.dll
*** WARNING: Unable to verify checksum for WindowsBase.ni.dll
*** WARNING: Unable to verify checksum for PresentationCore.ni.dll
*** WARNING: Unable to verify checksum for NextGen.Core.dll
MethodDesc:   45b52128
Method Name:  OnCallBackMethod(IntPtr, IntPtr, UInt32, ART_MARK_ATTRIBUTES, enumModifyReasons)
Class:        45b690fc
MethodTable:  45b52d70
mdToken:      06000242
Module:       1bcc94c4
IsJitted:     yes
CodeAddr:     09061a00
Transparency: Critical
Source file:  D:\Core\PasteImage.cs @ 724
MethodDesc:   476ed6a8
Method Name:  DomainBoundILStubClass.IL_STUB_ReversePInvoke(Int32, Int32, Int32, IntPtr, Int32)
Class:        07d51b3c
MethodTable:  07d51b90
mdToken:      06000000
Module:       00874044
IsJitted:     yes
CodeAddr:     47773910
Transparency: Safe critical
Failed to request MethodData, not in JIT code range

KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 67280

    Key  : Analysis.DebugAnalysisManager
    Value: Create

    Key  : Analysis.Elapsed.mSec
    Value: 75763

    Key  : Analysis.Init.CPU.mSec
    Value: 4436

    Key  : Analysis.Init.Elapsed.mSec
    Value: 9321

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 228

    Key  : CLR.BuiltBy
    Value: NET48REL1LAST_C

    Key  : CLR.Engine
    Value: CLR

    Key  : CLR.Version
    Value: 4.8.4515.0

    Key  : Timeline.OS.Boot.DeltaSec
    Value: 4888

    Key  : Timeline.Process.Start.DeltaSec
    Value: 117

    Key  : WER.OS.Branch
    Value: vb_release

    Key  : WER.OS.Timestamp
    Value: 2019-12-06T14:06:00Z

    Key  : WER.OS.Version
    Value: 10.0.19041.1

    Key  : WER.Process.Version
    Value: 6.0.0.209


FILE_IN_CAB:  CrashDump.dmp

NTGLOBALFLAG:  0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS:  0

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 00000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

FAULTING_THREAD:  00005474

PROCESS_NAME:  NextGenEMR.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE_STR:  80000003

IP_ON_HEAP:  477739d5
The fault address in not in any loaded module, please check your build's rebase
log at <releasedir>\bin\build_logs\timebuild\ntrebase.log for module which may
contain the address if it were loaded.

FRAME_ONE_INVALID: 1

STACK_TEXT:  
0019e0c8 09061cd1 OnCallBackMethod+0x2d1
0019e188 477739d5 unknown!unknown+0x0
0019e1ec 441b1553 unknown!unknown+0x0
0019e224 1c555afc igart18d+0x8e8c
0019e228 27195478 unknown!unknown+0x0
0019e240 27195478 unknown!unknown+0x0
0019e290 27195478 unknown!unknown+0x0
0019e2ac 1c5561d8 igart18d+0x9568


FAULTING_SOURCE_LINE:  D:\Core\PasteImage.cs

FAULTING_SOURCE_FILE:  D:\Core\PasteImage.cs

FAULTING_SOURCE_LINE_NUMBER:  724

SYMBOL_NAME:  OnCallBackMethod+2d1

MODULE_NAME: ImagingControl

IMAGE_NAME:  ImagingControl.dll

STACK_COMMAND:  .cxr 19d618 ; kb ; ** Pseudo Context ** Pseudo ** Value: f8b83f0 ** ; kb

FAILURE_BUCKET_ID:  BREAKPOINT_80000003_OnCallBackMethod

OS_VERSION:  10.0.19041.1

BUILDLAB_STR:  vb_release

OSPLATFORM_TYPE:  x86

OSNAME:  Windows 10

IMAGE_VERSION:  6.0.0.209

FAILURE_ID_HASH:  {bd0e81c1-c531-0b4a-964b-28450da23c4b}

Followup:     MachineOwner
---------


0:000> k
 # ChildEBP RetAddr      
00 0019d14c 75f82609     ntdll!NtWaitForSingleObject+0xc
01 0019d14c 729c1401     KERNELBASE!WaitForSingleObjectEx+0x99
02 0019d17c 729c145d     clr!CLREventWaitHelper2+0x33
03 0019d1cc 729c13aa     clr!CLREventWaitHelper+0x2a
04 0019d204 729c13cc     clr!CLREventBase::WaitEx+0x14b
05 0019d21c 72c9e065     clr!CLREventBase::Wait+0x1a
06 0019d288 72c9e17a     clr!Thread::WaitSuspendEventsHelper+0x8a
07 0019d29c 72c9c2aa     clr!Thread::WaitSuspendEvents+0x14
08 0019d2b4 72a89aae     clr!Thread::RareEnablePreemptiveGC+0x98
09 0019d2f8 72f642a5     clr!Thread::RareDisablePreemptiveGC+0x115
0a 0019d31c 72f69ccc     clr!GCHolderEEInterface<0,1,1>::~GCHolderEEInterface<0,1,1>+0x61
0b 0019d38c 72f6a75b     clr!DebuggerController::DispatchPatchOrSingleStep+0x1ef
0c 0019d3c0 72a7a625     clr!DebuggerController::DispatchNativeException+0x17c
0d 0019d3f0 72b4b82d     clr!Debugger::FirstChanceNativeException+0x69
0e 0019d430 72b4b91e     clr!CLRVectoredExceptionHandlerPhase2+0xa9
0f 0019d458 72b4d32c     clr!CLRVectoredExceptionHandler+0xac
10 0019d498 72b4d47a     clr!CPFH_FirstPassHandler+0x79
11 0019d4c4 77828a22     clr!COMPlusFrameHandler+0x15d
12 0019d4e8 778289f4     ntdll!ExecuteHandler2+0x26
13 0019d5b0 77814ee6     ntdll!ExecuteHandler+0x24
14 0019e170 09061cd1     ntdll!KiUserExceptionDispatcher+0x26
15 0019e170 477739d5     OnCallBackMethod+0x2d1 [PasteImage.cs @ 724] 
WARNING: Frame IP not in any known module. Following frames may be wrong.
16 0019e1e4 441b1553     0x477739d5
17 0019e21c 1c555afc     0x441b1553
18 27195478 27195478     igart18d!CPb_ART_done+0x8e8c
19 1c555afc 27195478     0x27195478
1a 0000000b 27195478     0x27195478
1b 1df00520 1c5561d8     0x27195478
1c 00000000 00000000     igart18d!CPb_ART_done+0x9568

0:000> dc 0x477739d5
477739d5  90909090 b9909090 46412a54 248baae8  ........T*AF...$
477739e5  c045892b ffc0458d c8fde830 45892b3f  +.E..E..0...?+.E
477739f5  b4558bb4 e80c4d8b 2b300d6f 000c7d83  ..U..M..o.0+.}..
47773a05  7d8b1974 b9c0330c 00000029 006aabf3  t..}.3..).....j.
47773a15  8bac4d8b 80e80c55 c72b300c 0004d845  .M..U....0+.E...
47773a25  90904000 d845c790 7fffffff 00e845c7  .@....E......E..
47773a35  c7000000 00fcec45 4f680000 eb47773a  ....E.....hO:wG.
47773a45  f8658d12 c25d5f5e 45c7000c 000000ec  ..e.^_]....E....

The application crash is random, sometimes it will crash, and sometimes it doesn't. The source code has not been changed for more than 2 years now.

I am assuming that it might be due to Heap Corruption, but need help in getting to understand if it is a Heap leakage or Heap Out of bounds exception. And if so, how to handle it.

raghava
  • 9
  • 5
  • 2
    Assuming you use Visual Studio for development, have you tried to use the VS debugger to catch the crash as it happens, using a debug-build of the DLL? – Some programmer dude Jul 14 '22 at 10:24
  • Yes. But it is not getting caught the only message I'm getting is "Unhandled exception at 0x77886FE3 (ntdll.dll) in Application.exe: 0xC0000374: A heap has been corrupted (parameters: 0x778C3960)." – raghava Jul 14 '22 at 10:29
  • Put lots of breakpoints and check which was the last breakpoint encountered. It will at least narrow down the code where you need to check. – kiner_shah Jul 14 '22 at 10:34
  • Tried that. The last breakpoint was the end brace of the CallBack function getting triggered. After the end brace, whatever I try, F5 F10, or F11, the application crashes. – raghava Jul 14 '22 at 10:36
  • Do the return of the function lead to any objects being destroyed? Are their destructors doing anything special? The C++ code doesn't go out of bounds of any arrays or memory? Have you tried stepping through the code itself to make sure it doesn't do anything bad? Have you tried removing parts of the code to try and pinpoint the crash? – Some programmer dude Jul 14 '22 at 10:38
  • Yes. I made sure that no objects are destroyed. One thing I would like to mention is that the same CallBack function is triggered various times and it is fine. The crash is only happening for a specific function callback. – raghava Jul 14 '22 at 10:40
  • 1
    And a small note about the term "exception". What happens here is a CPU or hardware exception. It's not related to C++ or C# exceptions, and can therefore not be caught by those languages in the normal way. – Some programmer dude Jul 14 '22 at 10:40
  • Then how about arguments, and everything else that can differ between other "working" invocations and this one that crashes? Also note, that in C++ you can have *undefined behavior* (like going out of bounds of allocated memory) and it might seem to work fine, just to have some unexpected behavior (like a crash) somewhere completely different. – Some programmer dude Jul 14 '22 at 10:42
  • So, how do I identify what is causing the crash? And how do I resolve it? – raghava Jul 14 '22 at 10:43
  • Also, are you using any kind of VCS (Version Control System, like e.g. Git)? If you rollback changes, does it start to work again? If so, what are the differences between the working commit and the non-working commit? It might not be the root cause (due to the unpredictable behavior of undefined behavior in C++), but could help narrow down and help to pinpoint the actual cause. – Some programmer dude Jul 14 '22 at 10:44
  • There were no changes in that particular project for the last 2 years. And the crash is random. Sometimes, it might not crash at all. – raghava Jul 14 '22 at 10:45
  • 2
    Yeah, that's typical UB. The only way is to try to remove C++ code bit by bit until you never have the crash, add code back in until you get the crash, and then step through the C++ code statement by statement to see what it does, while monitoring variables and their values. Unless you can show us a [mre], then there's really nothing more we can do you help you. – Some programmer dude Jul 14 '22 at 10:49
  • You should probably show some relevant code. At the very least, the code that invokes your callback, and your entire callback function. Even then, it could be pretty tricky to identify the issue here unless you're doing something obviously wrong. Delayed memory corruption issues can be quite challenging. Often they require invasive debugging techniques where you theorize what might be happening and construct methods for ruling that out as a possibility. If I was a gambling fella, I'd put money on a buffer overrun somewhere that stomps over some pointers held by the same or another object. – paddy Jul 14 '22 at 10:50
  • have you tried using the MS extension __try/exception in your dll ? https://learn.microsoft.com/en-us/cpp/cpp/try-except-statement?view=msvc-170 – AndersK Jul 14 '22 at 10:57
  • 3
    Just an idea : When you register the callback do you also pass C# data to the C++ code (as a pointer)? Anything the C++ code uses from the .Net side must be pinned. If not the garbage collector is free to move memory around... and then the C++ code will start writing into .Net memory where is shouldn't. Bottom line you should be very aware of the different lifecycle and memory models of C++ and .Net – Pepijn Kramer Jul 14 '22 at 11:00
  • Yes, from C#, I am passing IntPtr (Pointer) value to C++ Code. How do I pin that? – raghava Jul 14 '22 at 11:03
  • I haven't done it for a long time, but I have written C++/CLI wrappers around my C++ code and managed the stuff from there. But maybe this works too : https://stackoverflow.com/questions/37555360/how-can-i-pin-and-get-an-intptr-to-a-generic-t-array – Pepijn Kramer Jul 14 '22 at 11:09
  • I encountered one of these "works flawlessly for ages, then starts crashing seemingly at random" bugs and it was indeed a pinning issue. If that's the case, you may need to restructure some code in order to make sure that nothing moves around or becomes garbage until the C++ side is done with it, but it shouldn't be too hard to identify what data you need to pin down. (Doing it correctly might take some digging into details, though. You can't trust testing in this situation.) – molbdnilo Jul 14 '22 at 11:26

0 Answers0