1

i am kind of stumped on a memory leak in our application. It only happens on the customers machines, or at least tests to reproduce it on my machine have failed. Luckily our application collects some data to a log file in the event of a unhandled exception and writes a dump file if it is a OOM. It is a 32Bit C# WinForms program.

What i could find out so far:

  1. The process was running for about three hours before it crashed

  2. At the time of the crash the following values were collected:

currentProcess.WorkingSet64 = 1,8 GB currentProcess.VirtualMemorySize64 = 2.0 GB currentProcess.PrivateMemorySize64 = 680 MB

This is the first point that made me think it's unmanaged. Why is the WorkingSet so much higher that the PrivateMemory? I'd guess a third party dll maybe?

So for further analysis I loaded the memory dump into WinDbg

  1. The SOS command !analyzeoom tells me: There was no managed OOM due to allocations on the GC heap. To verify I had a look at !eeheap which tells me the GC heap has a total size of 217 MB and the Loader heap 38 MB

  2. Now i wanted to have a look at all the heaps. But using !heap -s showed a combined value of Commited Bytes 170 MB

  3. !dumpheap -stat shows again that the biggest consumers are around 10-25 MB. There are also around 4000 Free objects which make up 38 MB.

The last value made me think that maybe the OOM is caused by fragmentation. But 38 MB does not sound that much.

At this point I am not sure how to continue How do I locate this huge amount of missing memory? How can I find out if it is a third party assembly and which one?

Sadly I can't share the memory dump because it contains sensitive customer data. But I am grateful for any advice/tips

Thanatos
  • 41
  • 5
  • You'd better open a support case via http://support.microsoft.com to get assistance. Dump analysis isn't something trivial. – Lex Li Mar 17 '21 at 14:24
  • Thanks for the tip Li. I fear i might have to do that if i dont get anywhere soon But I am a step further i think. I got a new MemoryDump that was taken with more options enabled. Not i can see with the !address -summary command that indeed 1,236GB is occupied by .If i understood this right this means it's unmanaged memory (Images maybe?) and probably loaded by a third party dll. Any tips how to find out what type this unknown memory stuff is? – Thanatos Mar 18 '21 at 14:12
  • !heap -s is not showing any of the missing 1,2 GB. It all sums up to the size of the managed heap. There are no virtual blocks listed. – Thanatos Mar 18 '21 at 18:56
  • Found another clue: The command !GCHandleLeaks lists 2288 entries. That does not appear normal. What exactly is this output trying to tell me? I cant find much documentation on this command anywhere – Thanatos Mar 19 '21 at 09:10
  • `` could also be .NET memory. You have to rule that out by other means (like `!eeheap -gc` as you did). See https://stackoverflow.com/a/39428167/480982 – Thomas Weller Mar 22 '21 at 10:39
  • If you have a 32 Bit program, why are you looking at the `...64` values? Do these values have an actual meaning then, or do you then have a look at the WoW64 layer? – Thomas Weller Mar 22 '21 at 10:43
  • `!heap` is only helpful for allocations <512kB. Allocations larger than that are forwarded to `VirtualAlloc()`. Thus you won't see them in the output of `!heap`. – Thomas Weller Mar 22 '21 at 10:45
  • If your legal department prepares an NDA, I'm willing to sign it and I could have a look at the crash dump file. No cost but also no promises. You'll find contact information via my profile. – Thomas Weller Mar 22 '21 at 10:51
  • Thanks for the comments. 1) Yes. !eeheap -gc shows only ~180MB 2) According to the microsoft docs the Process.WorkingSet and so on are deprecated. Thats why i use the Process.WorkingSet64. Are thoose not correct for 32 Bit processes? 3)I understood that !heap -s shows virtual allocations as something like "Virtual block: 05c80000 - 05c80000 (size 00000000)" I dont get any of thoose in my dmp file Right now i am checking everywhere for Bitmaps that might not get disposed. – Thanatos Mar 23 '21 at 07:28
  • I saw the graph in the link you provided before. But i am not so sure how to decide with the few/may options. 1) large/small is 1,2 GB => large 2) few/many free objects => 324 objects => few i guess? 3) MEM_FREE = ~190 MB => Ordinary Out of Memory. – Thanatos Mar 23 '21 at 07:38
  • Ok, thanks for the update.It seems WorkingSet64 is fine. I just [wasn't sure](https://stackoverflow.com/a/36313379/480982). It may have been a bug in WinDbg or in the memory manager, but I'm quite sure I've had VirtualAlloc()s which didn't show up as virtual blocks. As you describe it, I'd follow the trace of !gchandleleaks – Thomas Weller Mar 24 '21 at 16:55
  • The "!gchandles" command tells me there are 19 Pinned Handles. I assume that would be "few" => Large Object Heap Fragmentation If i dump -stat the LOH i get: 59 Objects. 30 of them Free (11MB). That does not seem very concerning – Thanatos Mar 24 '21 at 20:11
  • I found a part in our application that uses a XmlSerializer which according to microsoft docs leaks memory because of dynamic generated assemblies. https://learn.microsoft.com/en-us/dotnet/api/system.xml.serialization.xmlserializer?view=netframework-4.8 Is that something that would show up as ? I tried to test it but did not get any good results – Thanatos Mar 24 '21 at 20:14
  • Best guess right now is that there were multiple factors. The XmlSerializer I mentioned in the comments, and also a PDF Viewer component with a rather unfortunate ImageCache configuration. – Thanatos Mar 31 '21 at 09:38

1 Answers1

0

It was a PDF Viewer component with a badly configured ImageCache Option. Thanks for all the help

Thanatos
  • 41
  • 5