We have migrated an ASP.NET 4 app to a new server (2008 R2) and are being plagued by nasty production issues - occasional stack overflows in the ASP.NET worker process.
These get logged in the event logs:
Faulting application name: w3wp.exe, version: 7.5.7601.17514, timestamp: 0x4ce7afa2
Faulting module name: clr.dll, version: 4.0.30319.269, time stamp: 0x4ee9d6e1
Exception code: 0xc00000fd
Fault offset: 0x0000000000003fbc
Faulting process id: 0x17ec
Faulting application start time: 0x01cdb8e2663f86bc
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Report Id: 45eb70a1-24da-11e2-a0b9-005056ba357d
Sometimes the faulting module is MSVCR100_CLR0400.dll, which is from the VC++ redistributable - but I don't think C++ has been used anywhere in the project
Windows has been kind enough to log a dump, which opened in Visual Studio. It complained with "no symbols loaded for w3wp.exe" so I only managed to get disassembly when I tried to debug and the call stack was just a single entry:
clr.dll!000007fef9afd01b()
That's in the case of the faulting module being clr.dll. In the other case of the exception, when the module is MSVCR100_CLR0400.dll:
MSVCR100_CLR0400.dll!000007fef9a3c9fa()
[Frames below may be incorrect and/or missing, no symbols loaded for MSVCR100_CLR0400.dll] MSVCR100_CLR0400.dll!000007fef9a57fcb()
MSVCR100_CLR0400.dll!000007fef9a33ee4()
MSVCR100_CLR0400.dll!000007fef9a33f5d()
clr.dll!000007fef9b05a08()
It looks fairly convincing that the fault is not coming from my code but rather is genuinly from within CLR / ASP.NET - I put together a test app that forces a stack overflow by infinite recursion. I get the dump, analyze it in VS and when I debug I hit the exact line of the recursive call.
On the other hand, the fault comes at a specific line in the application, so it's not some random effect - but it's not every time, just occasionally, and has never been reproduced outside the production environment.
It's also not a concurrency issue - it has been recorded with just 1 online user.
Needless to say, I've tried the basic troubleshooting techniques - try-catch everywhere, logging after each line, Application_Error - nothing. Execution just stops at some point and I get the error entry in Windows logs
App pool is in integrated mode, running under a service account created for this purpose.
