17

I'm currently working on a website that makes large use of cached data to avoid roundtrips. At startup we get a "large" graph (hundreds of thouthands of different kinds of objects). Those objects are retrieved over WCF and deserialized (we use protocol buffers for serialization) I'm using redgate's memory profiler to debug memory issues (the memory didn't seem to fit with how much memory we should need "after" we're done initializing and end up with this report

Global Report

Now what we can gather from this report is that:

1) Most of the memory .NET allocated is free (it may have been rightfully allocated during deserialisation, but now that it's free, i'd like for it to return to the OS)

2) Memory is fragmented (which is bad, as everytime i refresh the cash i need to redo the memory hungry deserialisation process and this, in turn creates large object that may throw an OutOfMemoryException due to fragmentation)

3) I have no clue why the space is fragmented, because when i look at the large object heap, there are only 30 instances, 15 object[] are directly attached to the GC and totally unrelated to me, 1 is a char array also attached directly to the GC Heap, the remaining 15 are mine but are not the cause of this as i get the same report if i comment them out in code.

So my question is, what can i do to go further with this? I'm not really sure what to look for in debugging / tools as it seems my memory is fragmented, but not by me, and huge amounts of free spaces are allocated by .net , which i can't release.

Also please make sure you understand the question well before answering, i'm not looking for a way to free memory within .net (GC.Collect), but to free memory that is already free in .net , to the system as well as to defragment said memory.

Note that a slow solution is fine, if it's possible to manually defragment the large heap i'd be all for it as i can call it at the end of RefreshCache and it's ok if it takes 1 or 2 second to run.

Thanks for your help!

A few notes i forgot: 1) The project is a .net 2.0 website, i get the same results running it in a .net 4 pool, idem if i run it in a .net 4 pool and convert it to .net 4 and recompile.

2) These are results of a release build, so debug build can not be the issue.

3) And this is probably quite important, i do not get these issues at all in the webdev server, only in IIS, in the webdev i get memory consumption rather close to my actual consumption (well more, but not 5-10X more!)

Ronan Thibaudau
  • 301
  • 2
  • 6
  • Is there any difference in processor architecture between the application pool on your server, and your dev server? – Rowland Shaw Mar 21 '12 at 13:01
  • I'm sorry my sentence wasn't very clear as i read it back, i didn't mean dev server as a separate server, but as in on the same server but running outside of iis (in asp.net developement server, integrated in visual studio 2010). – Ronan Thibaudau Mar 21 '12 at 13:21
  • For completeness (in both cases as it's the same machine) this is on a windows 2008 R2 X64 and the website is AnyCPU – Ronan Thibaudau Mar 21 '12 at 13:21
  • On that same subject, what about under IIS Express? I would expect the same behavior as IIS. Don't know that it's useful either. – Peter T. LaComb Jr. Mar 21 '12 at 13:33
  • I don't know but it wouldn't change much, i need it running under the real IIS in any case. I'm just hoping stating that asp.net dev server doesn't display this may help debug this situation. (note, i'd gladly test under iis express if that helps but i'm not sure i'm supposed to install it on a server that already has iis) – Ronan Thibaudau Mar 21 '12 at 13:36
  • Is the memory fragmentation due to native objects ? Do you have unmanaged objects ? – Seb Mar 21 '12 at 13:41
  • Have you heard of low fragmentation heap (LFH) ? – Seb Mar 21 '12 at 13:43
  • I don't have any unmanaged objects that i know of, but in any case it wouldn't be accounted for in that graph (the blue zone is only the free managed memory, there is a separate graph that indicates unmanaged memory and i sit at 100MB there, which is fine as i assume that's mostly IIS Dlls). I haven't heard of LFH, but as far as i get it, if there are 16 objects in the large heap, and none are owned by anyone but the GC, there shouldn't be any fragmentation, yet the memory profiler reports both fragmentation and lots of free memory that is kept by .net – Ronan Thibaudau Mar 21 '12 at 14:37

6 Answers6

9

Objects allocated on the large object heap (objects >= 85,000 bytes, normally arrays) are not compacted by the garbage collector. Microsoft decided that the cost of moving those objects around would be too high.

The recommendation is to reuse large objects if possible to avoid fragmentation on the managed heap and the VM space.

http://msdn.microsoft.com/en-us/magazine/cc534993.aspx

I'm assuming that your large objects are temporary byte arrays created by your deserialization library. If the library allows you to supply your own byte arrays, you could preallocate them at the start of the program and then reuse them.

Ben Challenor
  • 3,365
  • 1
  • 35
  • 35
8

I know this isn't the answer you'd like to hear, but you can't forcefully release the memory back to the OS. However, for what reason do you want to do so? .NET will free its heap back to the OS once you're running low on physical memory. But if there's an ample amount of free physical memory, .NET will keep its heap to make future allocation of objects faster. If you really wanted to force .NET to release its heap back to the OS, I suppose you could write a C program which just mallocs until it runs out of memory. This should cause the OS to signal .NET to free its unused portion of the heap.

It's better that unused memory be reseved for .NET so that your application will have better allocation performance (since the runtime knows what memory is free and what isn't, allocation can just use the free memory without having to syscall into the OS to get more memory).

The garbage collector is in charge of defragmenting the heap. Every so often (usually during collection runs), it will move objects around the heap if it determines this needs to be done. (This is why C++/CLI has the pin_ptr construct for "pinning" objects).

Fragmentation usually isn't a big issue though with memory, since it provides fast random access.

As for your OutOfMemoryException, I don't have a good answer for. Ordinarily I'd suspect that your old object graph isn't being collected (some object somewhere is holding a reference onto it, a "memory leak"). But since you're using a profiler, I don't know then.

atanamir
  • 4,833
  • 3
  • 24
  • 20
  • Your code is all managed C#, right? You don't do any pinning of pointers to marshal into unmanaged code? Otherwise, the runtime should defragment your memory for you. – atanamir Mar 21 '12 at 18:11
  • Nope, all managed code, also my application doesn't need better allocation performance, it needs to allocate about once a day, and it's a non blocking operation, the issue here is how much it overallocates (5X what's required) and memory fragmentation, afaik no the GC won't defragment the large object heap which is what causes OutOfMemoryExceptions, i've seen posts complaining about there being no way to triger a large object heap defrag too. The thing is once a day i refresh everything and getting an OutOfMemoryException every few days due to fragmentation is a big no no. – Ronan Thibaudau Mar 21 '12 at 20:49
  • You can try the simple C program that does a loop with a malloc(), breaking out once malloc() fails (the return value will be NULL). Then exit the program. That should force .NET to release all of its memory (but it will also make all processes do the same that's running on the OS too). – atanamir Mar 22 '12 at 05:08
  • But that wouldn't solve the fragmentation, and afaik the GC won't release fragmented bits, only segments that are fully freed, so while it may free some of the overallocated memory, i'll still be stuck with free fragmented memory that may add up to OutOfMemoryExceptions, I really wish there was a GC.CompactAllGen() – Ronan Thibaudau Mar 22 '12 at 10:53
  • If they're not large objects, the .NET runtime should rearrange the heap by moving everything to the beginning of the heap and then release the freed part of memory. Are you sure your profiler is showing a 'continual' fragmentation problem, or is it simply displaying the fragmentation at the time of the snapshot (e.g. between the runtime's defragmentation passes)? – atanamir Mar 24 '12 at 18:47
  • The profiler doesn't give me the detail (a continuous view of the memory would be nice) but it does say that memory fragmentation restricts the size of future allocations, which only makes sense in the context of the large object heap, on top of that it reports 15 fragments for 15 objects, which can't be good, but none of those are managed by me (they are GCRoot objects) – Ronan Thibaudau Mar 25 '12 at 23:31
4

As of .NET 4.5.1 you can set a one-time flag to compact LOH before issuing a call to GC collect, i.e.

Runtime.GCSettings.LargeObjectHeapCompactionMode = System.Runtime.GCLargeObjectHeapCompactionMode.CompactOnce; GC.Collect(); // This will cause the LOH to be compacted (once).

Shaun Landau
  • 51
  • 2
2

Some testing and some C++ later, i've found the reason why i get so much free memory, it's because of IIS instancing the CLR via VM Hoarding (providing a dll to instantiate it without VM Hoarding takes up as much initial memory, but does release most of it as time goes which is the behavior i expect). So this does fix my reported memory issue, however i still get about 100mb free memory no matter what, and i still think this is due to fragmentation and fragments only being released at once, because the profiler still reports memory fragmentation. So not marking my own answer as an answer in hope someone can shed some light on this or direct me to tools that can either fix this or help me debug the root cause.

Ronan Thibaudau
  • 301
  • 2
  • 6
  • I have exactly the same problem due to exactly the same reason. I also tracked it down to VM hoarding, although used WinDbg + SOS + SOSEX to inspect process dump and "Advanced .NET debugging" as a reference (it actually mentions hoarding as added .net 2 feature). That's were I am stuck for now. – bushed Aug 31 '12 at 11:09
  • One of the potential remedies I am considering might be spinning up separate process for precaching data from DB (we are using distributed cache) – bushed Aug 31 '12 at 11:10
  • Hey Ronan, I'm seeing what looks like to be the same exact problem with my application. You said VM Hoarding was the issue; how did you turn it off for IIS? Or how did you circumvent it another way? – Dave Sep 25 '12 at 17:47
  • Ronan, I have the same question as Dave. How did you turn it off for IIS or is it even possible? – Uchitha Mar 28 '14 at 05:18
1

It's intriguing that it works differently on the WebDevServer as to IIS...

Is it possible that IIS is using the server garbage-collector, and the WebDev server the workstation garbage collector? The method of garbage collection can affect fragmentation. It'll probably be set in your aspnet.config file. See: http://support.microsoft.com/kb/911716

Jason Crease
  • 1,896
  • 17
  • 17
-1

If you havent found your answer I think the following clues can help you :

Back to the basics : we sometimes forget that the objects can be explicitly set free, call explicitly the Dispose method of the objects (because you didnt mention it, I suppose you do an "object = null" instruction instead).

Use the inherited method, you dont need to implement one, unless your class doesnt have it, which I doubt it.

MSDN Help states about this method :

... There is no performance benefit in implementing the Dispose method on types that use only managed resources (such as arrays) because they are automatically reclaimed by the garbage collector. Use the Dispose method primarily on managed objects that use native resources and on COM objects that are exposed to the .NET Framework. ...

Because it says that "they are automatically reclaimed by garbage collector" we can infer that when the method is called does the "releasing thing" (Again Im trying only to give you clues).

Besides I found this interesting article (I suppose ... I didn read it ...completely) : Garbage Collection: Automatic Memory Management in the Microsoft .NET Framework (http://msdn.microsoft.com/en-us/magazine/bb985010.aspx) which states the following in the "Forcing an Object to Clean Up" section :

..., it is also recommended that you add an additional method to the type that allows a user of the type to explicitly clean up the object when they want. By convention, this method should be called Close or Dispose ....

Maybe the answer lies in this article if you read it carefully or just keep investigating in this direction.

fvalerin
  • 7
  • 1
  • 2
    When an object is freed by the garbage collector (including Dispose() or other means), the reclaimed memory goes into the "free space" heap of the process. That is the "blue" pie slice. The question is about releasing that memory back to the OS so other processes may use it. – crokusek Feb 19 '15 at 01:49