Chronic inappropriate System.OutOfMemoryException occurrences

Question

Here's a problem with what should be a continuously-running, unattended console app: I'm seeing too-frequent app exits from System.OutOfMemoryException being thrown from a wide variety of methods deep in the call stack -- often System.String.ToCharArray(), or System.String.CtorCharArrayStartLength(), or System.Xml.XmlTextReaderImpl.InitTextReaderInput(), but sometimes down in a System.String.Concat() call in a MongoCollection.Save() call's stack, and other unlikely places.

For what it's worth, we're using parallel tasks, but this is essentially the only app running on the server, and the app's total thread count never gets over 60. In some of these cases I know of a reason for some other exception to be thrown, but OutOfMemoryException makes no sense in these contexts, and it creates problems:

According to TaskManager and Perfmon logs, the system has had a minimum of 65% out of 8GB free memory when this has happened, and
While exception handlers sometimes fire & log the exception, they do not prevent an app crash, and
There's no continuing from this exception without user interaction (unless you suppress windows error reporting, which isn't what we want system-wide, or run the app as a service, which is possible but sub-optimal for our use-case)

So I'm aware of the workarounds mentioned above, but what I'd really like is some explanation -- and ideally a code-based handler -- for the unexpected OOM exceptions, so that we can engage appropriate continuation logic. Any ideas?

Memory fragmentation might go some way to explaining what's going on. Large, long lived allocations can reduce contiguous memory space well before total memory is exhausted. — spender, Jul 07 '12 at 01:24

score 2 · Accepted Answer · answered Jul 07 '12 at 01:04

2

Getting that exception when using under 3GB of memory suggests that you are running a 32-bit app. Build it as a 64-bit app and it will be able to use as much memory as is available (close to 8GB).

As to why it's failing in the first place...how large is the data are you working with? If it's not very large, have you looked for references to data being kept around much longer than they are necessary (i.e. a memory leak), thus preventing proper GC?

answered Jul 07 '12 at 01:04

Tim S.

55,448
7
96
122

1

Actually, I'd recommend building as AnyCPU if possible - no reason to prevent it from being used on an x86 machine unnecessarily. But yes, I'm guessing that's what the problem is. – Jonathan Rupp Jul 07 '12 at 01:07
The crashing version actually is being built as 'Any CPU', but out of curiosity I'll build it as x64 and see what happens. – Paul Smith Jul 07 '12 at 03:07
Well butter my buns and call me a biscuit... built as Any CPU sure enough I get the OutOfMemoryException; built as x64, not a one. I thought Any CPU allowed for > 2GB access, but apparently I missed something. – Paul Smith Jul 07 '12 at 03:09
I suspect you will get the exceptions eventually depending on length of time running. – AMissico Jul 07 '12 at 06:53
There is a setting called "Prefer 32-bit" since VS2012, make sure to DISABLE that. – MichelZ Nov 02 '16 at 15:05

score 2 · Answer 2 · answered Jul 07 '12 at 01:07

2

You need to profile the application, but the most common reason for these exceptions is excessive string creation. Also, excessive serialization can cause this and excessive Xslt transformations.

answered Jul 07 '12 at 01:07

AMissico

21,470
7
78
106

Excessive `string` and/or `char[]` creation seems likely considering his example locations where it errored. Good point. – Tim S. Jul 07 '12 at 01:12
I'm sure I can find plenty of inefficiency to remedy in the app, but I was hoping for the lazy way out below. :) It's wasteful as all getout, indeed, but given 4 gigs or so it does manage to run & exit as planned without issues. – Paul Smith Jul 07 '12 at 03:19
I think the biggest bang for your buck is to look for objects where `Dispose` is not being called. Next look for excessive string handling in loops. You might even benefit from running Code Analysis with the "performance" rule set. – AMissico Jul 07 '12 at 06:57

score 1 · Answer 3 · edited May 23 '17 at 11:50

Do you have a lot of objects larger or equal to 85000 bytes? Every such object will go to the Large Object Heap, which is not compacted. I.e. unlike Small Object Heap, GC will not move objects around to fill the memory holes, which can lead to fragmentation, which is a potential problem for long-lived applications.

As of .NET 4, this is still the case, but it seems they made some improvements in .NET 4.5.

A quick-and-dirty workaround is to make sure the application can use all the available memory by building it as "x64" or "Any CPU", but the real solution would be to minimize repeated allocation/deallocation cycles of large objects (i.e. use object pooling or avoid large objects altogether, if possible).

You might also want to look at this.

Chronic inappropriate System.OutOfMemoryException occurrences

3 Answers3