2

We have a large ASP.NET application that occasionally crashes due to StackOverflowExceptions. Because these aren't handled very elegantly by .NET, we are reduced to post-mortem debugging without any of our normal exception logs and stack traces. Once we find where the issue is occurring it is generally quite easy to fix; the hard part is pin-pointing where in the codebase the error happens.

The process dump file that we get after the crash seems like it would help greatly in this effort, but thus far, we've been unable to figure out how to best use it. You can (very, very, slowly) "debug" the process using visual studio, but this basically takes forever loading up MSFT symbols and then won't load the symbols for our application DLLs (so you can't see the interesting parts of the call stack).

It seems like there must be a straightforward way to go from:

  • The crash dump file
  • The set managed application DLLs/PDBs

to the full managed call stack; can anyone can describe (or point to a tutorial) for doing this (using VS, WinDbg, or any other tool)?

Community
  • 1
  • 1
ChaseMedallion
  • 20,860
  • 17
  • 88
  • 152
  • I'm curious to know what kind of areas you find these exceptions in. I don't recall ever finding a StackOverflowException in ASP.NET. – John Saunders May 06 '14 at 22:18
  • @JohnSaunders: one example we saw was a developer combining a bunch of lists using Aggregate and Concat instead of SelectMany. The most recent case seems to be something with a very complex query causing the EF expression visitor to overflow the stack (I suspect a large IN clause). – ChaseMedallion May 06 '14 at 22:23
  • Wow. Thanks for the answer. These could probably not be caught in unit testing, and I bet a try/catch wouldn't have caught them. – John Saunders May 06 '14 at 23:02
  • `won't load the symbols for our application DLLs` <-- why not? Open the `Debug -> Windows -> Modules` window in VS and see why your pdbs arent being found.(right click on your dll and choose 'Symbol Load Information') I reckon you are doing it a good way (MSFT symbols will be cached and loaded more quickly after the first time) – wal May 06 '14 at 23:45

2 Answers2

2

Yes, there is a way to get the managed stack, and more..

What you are looking for is the SOS.dll debugger extension for WinDbg. The SOS.dll debugger extension is located in the same folder where the .NET framework is installed (ie. C:\Windows\Microsoft.NET\Framework\v4.0.30319\SOS.dll)

Once you load SOS.dll into WinDbg you can query for:

  • managed call stack: !clrstack
  • managed threads: !threads
  • stack (local) variables !clrstack –l
  • function call arguments !clrstack -p
  • information about an object !do address
  • memory consumption !dumpheap
  • object holding reference to a memory address !gcroot address
  • managed locks !syncblk

Here is a cheat-sheet with commands to get around using sos.dll.

Mike Dinescu
  • 54,171
  • 16
  • 118
  • 151
  • This sounds great. As I've never used WinDbg or SOS, would you mind posting a sample sequence of commands to get to the callstack? – ChaseMedallion May 06 '14 at 22:20
  • I'm running a bit late but if you do a google search for how to debug with WinDbg and SOS.dll I'm sure you'll find a few tutorials.. that said, you may be able to capture these exceptions and generate some stack traces which will eliminate the need from using WinDbg + SOS.dll which albeit more powerful can be a tedious process.. – Mike Dinescu May 06 '14 at 22:24
  • How do I capture the exception and generate traces without WinDbg? – ChaseMedallion May 06 '14 at 22:25
  • @ChaseMedallion - you're right.. please disregard the comment about the stack traces.. I overlooked the fact that you were dealing with StackOverFlowException. But you should be able to use the SOS.dll extension to troubleshoot. – Mike Dinescu May 06 '14 at 23:12
0

You should get quite far with this approach:

Make sure you have the correct symbols

.symfix d:\symbols

Load the .NET extension in WinDbg

.loadby sos clr
.loadby sos mscorwks

Select the thread with the exception

~#s

Print the exception

!pe

and the callstacks

!clrstack
!dumpstack

If you have trouble with the version of SOS or mscordacwks, because the dump is not from the same machine, get the correct version from the server using Mscordacwks Collector.

Also note that there is always one StackOverflowException pre-allocated, but not necessarily thrown. The output of a command like

!dumpheap -stat -type StackOverflowException

might be misleading.

Thomas Weller
  • 55,411
  • 20
  • 125
  • 222