5

My IIS app (.NET 4.0, IIS7) is continually growing in memory, and eventually falls over, as if I have a memory leak.

So I took a DMP while it was up around 1.7GB and cracked it open in WinDbg.

The dumpheap -stat command showed that while I had a fair amount of objects in the heap, the majority (and bit that is growing when looking at sequential DMPs) is labelled as "Free" (>800MB):

000007feef55b768 46091 2212368 System.Data.DataRowView
000007fe9739dda8 10355 2236680 Newtonsoft.Json.Serialization.JsonProperty
000007fef4260610 33062 2644960 System.Signature
000007fef4242250 41809 4682608 System.Reflection.RuntimeMethodInfo
000007fef424f058 69232 8847997 System.Byte[]
000007fef4241b28 11 9437680 System.Double[]
000007fef4237ca0 15 9505176 System.DateTime[]
000007fef424c168 32559 11009308 System.Char[]
000007fef424dc30 17271 11795376 System.Int32[]
000007feef555c48 908 17936672 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
000007feef554f58 853 22056952 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]][]
000007feef5514b0 541444 51978624 System.Data.DataRow
000007fef424aee0 1550958 132599872 System.String
000007fef422f1b8 183607 178409288 System.Object[]
0000000000d8b2d0 234017 844500226 Free

So I then ran "!dumpheap -type Free" which gave me a lot of tiny Free objects (94 bytes each to be exact!) and towards the end a couple of larger "Free" objects:

00000003098a59b0 0000000000d8b2d0 134 Free
00000003098c19d0 0000000000d8b2d0 102 Free
00000003098ffa00 0000000000d8b2d0 54 Free
0000000309a41d98 0000000000d8b2d0 5961750 Free
000000041f8a1000 0000000000d8b2d0 24 Free
000000042001b4d0 0000000000d8b2d0 16933078 Free
00000004211bf7c8 0000000000d8b2d0 7702 Free
00000004212c1600 0000000000d8b2d0 35173374 Free
00000004236b3be0 0000000000d8b2d0 66886 Free
0000000423cc41e8 0000000000d8b2d0 10778318 Free
0000000424768928 0000000000d8b2d0 2254734 Free
00000004249ec128 0000000000d8b2d0 21166350 Free
000000042600f1e0 0000000000d8b2d0 51366 Free
000000042621bac8 0000000000d8b2d0 114007238 Free

Statistics:
              MT    Count    TotalSize Class Name
000007fef31e7460        1           32 System.Net.SafeLocalFree
0000000000d8b2d0   234017    844500226      Free
Total 234018 objects

Noting the repeating address of "d8b2d0" I ran a "!gcroot d8b2d0" to which I got the following result:

0:000> !gcroot d8b2d0
Found 0 unique roots (run '!GCRoot -all' to see all roots).

So... after all that... I have masses of Free objects stuck on my heap that the GC is not releasing. This builds up over about 2-3 hours. There is no other sign of any memory leaks from actual typed objects. It happens over almost all of our VMs in production.

Does anyone have any idea how to deal with the GC leaving so much Free object space on the heap?

It's had me running in circles for days. So hat's off to the genius that can help me figure this one out!

trincot
  • 317,000
  • 35
  • 244
  • 286
Omar McIver
  • 111
  • 6
  • Is this an Asp.Net application? Any particular framework or library used? – ljubomir May 14 '15 at 09:31
  • yes it's asp.net. It's mostly DataTables and some local WCF using net.pipe. – Omar McIver May 14 '15 at 09:44
  • Do you have any interop code or code that allocates `GCHandle`s directly? I'm thinking that maybe you have a lot of *pinned* objects that are preventing compaction. – Damien_The_Unbeliever May 14 '15 at 10:00
  • I think this could be the issue, but I'm determin to discover exactly what the source is... the Datatables (or more specifically, a certain way of using then) or WCF or something else!? https://www.simple-talk.com/dotnet/.net-framework/large-object-heap-compaction-should-you-use-it/ – Omar McIver May 14 '15 at 10:01
  • No interop code I'm aware of. Plus if it was pinned, wouldn't the gcroot find a root? – Omar McIver May 14 '15 at 15:17
  • 1
    use PerfView and select "Take Heap Snapshot from dump". Now open the generated gcdump file in PerfView to see a nicer overview: http://www.microsoft.com/en-us/download/details.aspx?id=28567 – magicandre1981 May 15 '15 at 06:02

2 Answers2

2

Thanks to the suggestion on using PerfView by magicandre1981, and after quite a few very large memory dumps, I've located this issue down to the use of DataRow.DefaultIfEmpty() in a Linq expression. It was part of a support function for carrying out a LeftJoin between two DataTables.

With the DefaultIfEmpty() in the DataRow enumeration as part of the Linq expression, I could recreate memory sky rocketing to 4GB+ in minutes. Most of this survives through to the Gen2 GC collection through sheer volume of ItemArrayObjects being created (both small and large object heap). This in turn lead to an ever climbing committed memory value. The GC2 collection just can't keep up on the small heap, and the large heap suffers from poor (if any) reuse.

We've taken this call out and the same recreation never gets more than 50MB in committed memory. We now need to see what the impact of this is on our DataTable joins, but the memory heap mystery has been solved!

Thanks for the help folks.

Community
  • 1
  • 1
Omar McIver
  • 111
  • 6
  • Can you state the steps you took to reach that conclusion? I'm struggling to find a cause in a very much similar situation. – Nuno Miguel Fonseca Jul 25 '18 at 16:06
  • Start by downloading and learning how to take memory dump files for managed code. There is a more extensive setup for triggering them as and when crashes happen or performance limits are exceeded, but I just attached the debugger to my app when it started (and then again when it inflated) and created snapshot memory dumps. https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-download-tools – Omar McIver Jul 26 '18 at 17:20
  • Then, I loaded the memory dumps in to PerfView (as noted above) and it was easy to browse around the objects. Comparing the first snapshot with the latter you start to see a pattern of certain types of objects causing the inflation in memory. – Omar McIver Jul 26 '18 at 17:24
  • Somewhere on the web I found guidance on what type of objects cause the most 'survival' issues where they persist in to GC2 even when no longer believed to be in use. Well I found that when you use the DefaultIfEmpty on a DataRow, it creates an interim ItemArrayObjects object which never gets marked as dead as it is associated with the DataRow, in turn the DataTable and in turn the DataSet - so until I dispose of my dataset, that call will generate more surviving garbage. – Omar McIver Jul 26 '18 at 17:24
  • As for finding that hot spot in your code, it took some trial and error to remove calls and isolate the inflation. I would say it was identifying that the ItemArrayObjects were the source of the garbage that led me to find that the DefaultIfEmpty() call was the likely culprit, as I wasn't doing anything in the code myself with respect to that type of object, ergo, it must be something DataRow related. – Omar McIver Jul 26 '18 at 17:27
  • Thank you for the detailed steps. In my case i definitely see a kind of objects that seem responsible for the inflation - and they belong to a structure that is static, so it will not go anywhere, although the objects themselves may be released - but the worst is really the "free" space, in my case is above 1GB. In your case you saw a correlation between the heavy use of certain objects and heap fragmentation. I'm not really sure that's the case in my example. Thanks again! – Nuno Miguel Fonseca Jul 30 '18 at 13:42
0

If you are certain that your app has memory leak i would suggest that you start local analysis with memory profiler. This might reveal the source of the problem faster than analyzing memory dumps in production.

ljubomir
  • 1,495
  • 1
  • 21
  • 40
  • Thanks for the suggestion. I've been through all that. There are no memory leaks, which is what has lead me to investigate a memory DMP file instead. From the analysis above, it comfirms (I believe) that indeed there isn't a memory leak. Otherwise that large 800MB+ chunk would have a gcroot or at least an object type of some sort. Instead it's actually labelled as *free* but the GC just won't release it!? – Omar McIver May 14 '15 at 09:37