0

Given two PCs

  1. Core i5 3rd Gen 4GB RAM (Dell Optiflex 7010)
  2. Core i7 2nd Gen 16GB RAM (Samsung 700G7C)

Update 1 - what's interesting, reducing the number of T's properties by two times changes nothing, the number of Ts in the collection at which OOM happens remains the same...

take the average low frequencies, I just don't have them to hands.

Can anyone explain to me, why the code working perfectly fine on the first PC fails 100% of times on the second one with OutOfMemoryException? It's really annoying considering that the second PC cost is 3 times higher.

I can't post the entire code here, but it's really simple - a List gets populated with T's, where T is a CLR object with 5 Integer properties. First PC handles 25M objects no problem (that's what I saw, it could probably handle even more) whilst the second one dies at approx. 16.5M records. I am aware of a single object memory limit, but what it really gets me is how it can deviate so much (50%++) between two quite modern PCs?

user1514042
  • 1,899
  • 7
  • 31
  • 57
  • 1
    Same version of Windows (32bit/64bit)? What about the virtual memory size? It matters more how big your page file can get than how much RAM you have. – jlew Apr 15 '13 at 17:38
  • http://stackoverflow.com/questions/8563933/c-sharp-out-of-memory-exception – BNL Apr 15 '13 at 17:39
  • 3
    one running 64-bit and another running 32-bit is a usual culprit. – Peter Ritchie Apr 15 '13 at 17:39
  • Win7 64bit with latests updates, I haven't checked the virtual memory size, how do I do that? – user1514042 Apr 15 '13 at 17:42
  • @Peter, I'm 90% sure the both run on 64bit platform, The second PC for definite, while it demonstrates worse performance. – user1514042 Apr 15 '13 at 17:43
  • @user1514042 You can check for sure with `Environment.Is64BitProcess` – Peter Ritchie Apr 15 '13 at 17:47
  • If one is Server and the other isn't, that can sometimes produce different results. Each uses a different GC algorithm which could lead to allocating memory differently and result in less-space-efficient allocations in one case and not the other. – Peter Ritchie Apr 15 '13 at 17:48
  • Is `T` a class or a struct? If it is a class then the array inside `List` will be able to hold fewer refs on 64 bit than on 32 bit due to the single object memory limit. – Brian Rasmussen Apr 15 '13 at 17:49
  • @Peter both are Win7 workstations – user1514042 Apr 15 '13 at 17:53
  • @Brian Rasmussen classes. – user1514042 Apr 15 '13 at 17:53
  • This may be relevant: http://blogs.msdn.com/b/joshwil/archive/2005/08/10/450202.aspx – Brian Rasmussen Apr 15 '13 at 17:57
  • @Brian Rasmussen thanks, I saw that link, I was kinda hoping to find the initial reason. A lot of things seem to be pear shaped on the second PC - for instance SqlBulkLoad for the same file fails on the second PC only, I had to split by two portions and so on. – user1514042 Apr 15 '13 at 18:02

3 Answers3

3

The question is heavily under-documented, but I can reverse-engineer some of it. First off, the only way you can run out of memory is when you use a list of structures instead of classes. That makes a structure size 4 * 5 = 20 bytes. At 16.5 million elements, you'll need an internal array for the List<> that has at least 20 x 16.5 = 330 megabyte of contiguous virtual memory. The next allocation will ask for double the size, 660 megabytes. That's hard to come by in a 32-bit process. The risk of OOM is very close with that size.

At issue is that you need a continguous allocation. In other words, a unused hole in the virtual memory address space that is at least 660 megabytes. The trouble is that the VM space needs to be shared by both code and data. And code is the usual troublemaker here. A DLL that gets loaded on one machine but not the other. Like the shovelware from Intel, the vendor or a virus scanner. A DLL has a preferred load address that can be very awkward, splitting the available address space into two smaller parts. The sum of the parts still plenty large enough but not leaving a big enough hole to fit a 660 MB allocation.

This is a general problem called "address space fragmentation". It is always the first cause of OOM, it is very hard to consume all 2 gigabytes of available address space. That could only happen by making very small allocations.

Several very simple things you can do to fix this problem:

  • Get rid of the shovelware on that machine, use SysInternals' VMMap utility to see if that worked
  • Use a class instead of a structure. Now the List element size is only 4 bytes instead of 20
  • Use the List<>.Capacity property. This can reduce the address space fragmentation you get from having the internal array reallocated several times while the list grows. As long as you have a good initial guess.
  • You've got nice hardware, use it. Change the Target platform setting on the EXE project to AnyCPU. A 64-bit process has massive amounts of address space, very hard to consume it all.
  • You can run Editbin.exe with the /LARGEADDRESSAWARE option on your EXE. Now your 32-bit process will have a 4 gigabyte address space on a 64-bit operating system.
Hans Passant
  • 922,412
  • 146
  • 1,693
  • 2,536
1

This may sound strange but easy to rule out. Use a memory profiler and watch for source of leak.

The only way I can figure out a low spec hardware survive and a high one fails is memory fragmentation. GC is not good at this.

Xaqron
  • 29,931
  • 42
  • 140
  • 205
  • I run my tests with no other apps are running, so in theory fragmentation should not be an issue. Could I use ANTS memory profiler on both machines to identify the reason for that huge performance disparity on those two machines? Could you suggest what should I look at first? – user1514042 Apr 21 '13 at 11:49
  • @user1514042: Anyway, without any clue, first step would be memory profiling. ANTS has a trial period to use free so you should frame the problem in that period or buy the license to continue using it. So be fast when installed. Installing on problematic machine is sufficient unless you are looking for special patterns. Post result here so we can help more. – Xaqron Apr 21 '13 at 12:47
0

Seting build platform to x64 (by default it's set to Any Platform) has fixed my problem. I've tried a good few recipes, including gcAllowVeryLargeObjects and forcing x86, but to no avail.

user1514042
  • 1,899
  • 7
  • 31
  • 57