1

I'm doing postmortem analysis on a high memory pressure .net application using WinDbg with a process dump and this process is a Windows service.

I got a feeling that most of this 14GB of this process memory consumption comes from aborted threads thus lots of orphan semaphore/events/mutant etc. But I wasn't able to put all this together and add them up like how much memory it takes for a single semaphore/events/mutant and what kind of WinDbg commands that will be helpful for this kind of situation?

The following is the WinDbg output:

!handle

**Type                      Count**

None                        90
Event                       5550
Section                     41
File                        1166
Directory                   3
Mutant                      160
Semaphore                   4581
Key                         78
Token                       2
Thread                      553
IoCompletion                6
Timer                       1
TpWorkerFactory             3
ALPC Port                   9
WaitCompletionPacket        33

!address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal

<unknown>                               471        3`86ea2000 (  14.108 Gb)  92.40%    0.01%

!threads(lots of thread listed have ThreadAbortException exceptions)

Lock  
   ID OSID ThreadOBJ           State GC Mode     GC Alloc Context                  Domain           Count Apt Exception

  12    3 33f0 00000017e3c23200 1282b221 Preemptive  0000000000000000:0000000000000000 00000017e3bb3930 1     MTA System.Threading.ThreadAbortException 000000181de5d668
sophia liu
  • 91
  • 1
  • 5
  • Might also be helpful: https://stackoverflow.com/questions/26142607/how-to-use-windbg-to-track-down-net-out-of-memory-exceptions – Thomas Weller Mar 15 '16 at 10:32
  • With *that* many unfinalized OS resources it is little wonder that you see high memory usage. A common cause is a dead-locked finalizer thread, once that happens memory usage climbs without bounds. Find it back in the minidump and look at its callstack. – Hans Passant Mar 15 '16 at 11:33
  • @HansPassant It seems the finalizer thread did get blocked from the following output but i didn't see anywhere in the code where a finalizer exists:0:000> ~[2]k Child-SP RetAddr Call Site 00000017`fd5bf2c8 00007ffc`d340dd29 ntdll!NtWaitForSingleObject+0xa 00000017`fd5bf2d0 00007ffc`d340b6f4 ntdll!RtlpWaitOnCriticalSection+0xe1 – sophia liu Mar 21 '16 at 09:26
  • It is almost always blocked, what matters is whether it is *dead-locked*. You did not post nearly enough of the stack trace for us to see that. Consider calling Microsoft Support to get help with this. – Hans Passant Mar 21 '16 at 09:38

1 Answers1

2

Handles and their wrappers are likely quite small. Use dt -r to get an idea of their size:

0:025> dt -r ntdll!_KSEMAPHORE
   +0x000 Header           : _DISPATCHER_HEADER
      +0x000 Type             : UChar
      +0x001 TimerControlFlags : UChar
      +0x001 Absolute         : Pos 0, 1 Bit
      +0x001 Coalescable      : Pos 1, 1 Bit
      +0x001 KeepShifting     : Pos 2, 1 Bit
      +0x001 EncodedTolerableDelay : Pos 3, 5 Bits
      +0x001 Abandoned        : UChar
      +0x001 Signalling       : UChar
      +0x002 ThreadControlFlags : UChar
      +0x002 CpuThrottled     : Pos 0, 1 Bit
      +0x002 CycleProfiling   : Pos 1, 1 Bit
      +0x002 CounterProfiling : Pos 2, 1 Bit
      +0x002 Reserved         : Pos 3, 5 Bits
      +0x002 Hand             : UChar
      +0x002 Size             : UChar
      +0x003 TimerMiscFlags   : UChar
      +0x003 Index            : Pos 0, 6 Bits
      +0x003 Inserted         : Pos 6, 1 Bit
      +0x003 Expired          : Pos 7, 1 Bit
      +0x003 DebugActive      : UChar
      +0x003 ActiveDR7        : Pos 0, 1 Bit
      +0x003 Instrumented     : Pos 1, 1 Bit
      +0x003 Reserved2        : Pos 2, 4 Bits
      +0x003 UmsScheduled     : Pos 6, 1 Bit
      +0x003 UmsPrimary       : Pos 7, 1 Bit
      +0x003 DpcActive        : UChar
      +0x000 Lock             : Int4B
      +0x004 SignalState      : Int4B
      +0x008 WaitListHead     : _LIST_ENTRY
         +0x000 Flink            : Ptr64 _LIST_ENTRY
         +0x008 Blink            : Ptr64 _LIST_ENTRY
   +0x018 Limit            : Int4B

So that Semaphore is 0x18+4 bytes in size. Be aware that the size might increase if the Ptr parts are used, e.g. if you have a very large wait list.

I recommend a !dumpheap -stat in your situation. This will output a list of .NET objects, sorted by their total size.

Typically there are byte[], object[] (or similar) and String which eat most memory.

0:025> !dumpheap -mt 000007feef0aea80
         Address               MT     Size
[...]
000007feef0bda88     4915       329862 System.String
000007feef0be100     1419       382288 System.Object[]
000007feef05f748        2       786520 System.UInt32[]
Total 65575 objects

For a single object, you can use !objsize <address>. A System.Threading.Thread in my example had 16 kB. So even if you have 1000 threads, this would just make up for 16 MB of memory. An example of a System.Threading.ManualResetEvent accounts for only 80 bytes in my application.

0:025> !objsize 0000000011878358 
sizeof(0000000011878358) = 16736 (0x4160) bytes (System.Threading.Thread)

0:025> !objsize 00000000118af810 
sizeof(00000000118af810) = 80 (0x50) bytes (System.Threading.ManualResetEvent)
Thomas Weller
  • 55,411
  • 20
  • 125
  • 222