Getting OutOfMemoryException in unit test

Question

This a moot question, as the amount I was allocating was wrong. It was supposed to be 400MB not 400GB... Sorry! The answers are good though.

I am trying to unit test a method of mine which throws an OutOfMemoryException if a large file is uploaded.

My problem is my unit test is also throwing this exception (for the same reasons I guess).

According to Task Manager, if I am reading the information correctly, I do have enough RAM so am not really sure why my unit test is failing in this way.

RAM usage

Here is the respective unit test code:

var data = new List<byte>();
for(UInt64 i = 0; i < 400ul * 1024ul * 1024ul * 1024ul; i += 65536ul)
{
    data.AddRange(new byte[65536]); // <-- throws exception here when i = 536870912
}
this.UploadedFileData = data.ToArray();

I tried the following before the above code:

var data = new byte[400ul * 1024ul * 1024ul * 1024ul];

This though, results in the error: Arithmetic operation resulted in an overflow.

This is an utterly preposterous way to accomplish what you are trying to do! Do you understand what [AddRange](http://msdn.microsoft.com/en-us/library/z883w3dc.aspx) does? Did you read this: "If the new `Count` (the current `Count` plus the size of the collection) will be greater than `Capacity`, the capacity of the List is increased by **automatically reallocating the internal array to accommodate the new elements, and the existing elements are copied to the new array** before the new elements are added." Nuke this code from orbit and write something sensible! — David Schwartz, Jan 11 '13 at 06:24
Yes. The above code is simple to create a really big, empty array, which will pretend to be a really big uploaded file. What do you suggest? — rhughes, Jan 11 '13 at 06:26
For one thing, allocate *all* the memory you need at once and *then* fill it in. And don't assume you can store that many bytes contiguously. That code is flat out nuts. Seriously. This is the worst example of the 'repeated string concatenation' anti-pattern I've seen in some time. You have **6 million** huge allocations, copies, and frees when you actually need ... **one**. Memory is going to be a *mess* halfway into that. — David Schwartz, Jan 11 '13 at 06:26
"Out of Memory" has nothing to do with the amount of memory on your machine. It usually refers to address space. — Damien_The_Unbeliever, Jan 11 '13 at 06:30
`var data = new byte[400ul * 1024ul * 1024ul * 1024ul];` You're trying to allocate 400 *gigabytes*. (!) — aquinas, Jan 11 '13 at 06:45

score 4 · Accepted Answer · edited May 23 '17 at 11:44

4

Are you on a 32 bit machine?

The process limit is 2GB on 32-bit, unless you use the /3GB boot switch.

From C# Increase Heap Size - Is It Possible

It is possible that you need to do the below

For very large List objects, you can increase the maximum capacity to 2 billion elements on a 64-bit system by setting the enabled attribute of the gcAllowVeryLargeObjects configuration element to true in the run-time environment.

From http://msdn.microsoft.com/en-us/library/6sh2ey19.aspx

As David mentions, the code will require a lot of reallocations when it grows. To prevent performance costs and memory fragmentation, you should at the very least set Capacity to an appropriate amount, if not rewrite the code.

edited May 23 '17 at 11:44

Community

1
1

answered Jan 11 '13 at 06:20

Karthik T

31,456
5
68
87

The machine is x64. I know VS is 32-bit though. Do you think this could be the issue? – rhughes Jan 11 '13 at 06:21
x64 memory addressing should not be available for 32-bit process. – abatishchev Jan 11 '13 at 06:24
@rhughes I am not sure, it is also very possible that the fragmentation caused by what David mentions is causing the exception. – Karthik T Jan 11 '13 at 06:29
If you're trying to get demons out of your grandmother's head with a jackhammer, you should just stop doing it. It doesn't matter precisely why it won't work. And it's not much of a victory if you finally do manage to get it to work because it's not sensible to do anyway. – David Schwartz Jan 11 '13 at 06:31

score 1 · Answer 2 · edited May 23 '17 at 11:44

Most likely you are hitting maximum allocation size limit (2GB for most versions) and it needs to throw something - OOM is perfect for that.

Limits are covered here: Single objects still limited to 2 GB in size in CLR 4.0?

Side note: MemoryStream and List are bad choices when you want slowly grow them to huge size due to need to store underlying data in continous block of memory. Either pre-allocate with necessary amount of storage or use custom chunked stream/list.

Sample of array that get around 2Gb allocation limit in pre 4.5 CLR - http://blogs.msdn.com/b/joshwil/archive/2005/08/10/450202.aspx

Getting OutOfMemoryException in unit test

2 Answers2