System.OutOfMemoryException while working with large Lists

Question

I have a code:

this.weights_StoA = new List<List<double>>();

if (NETWORK_MODE == 0)
{
    Random rand = new Random();

    int count = enters.Count;

    Parallel.For(0,  HIDDEN_NEURONS_COUNT, (i, loopState) =>
    {
        List<double> weights = new List<double>();

        for (int j = 0; j < count; j++)
        {
            weights.Add(rand.NextDouble());
        }

        lock (weights_StoA)
        {
            weights_StoA.Add(weights);
        }
    });
}

weights_StoA is a List<List<double>>.

I working with large arrays. HIDDEN_NEURONS_COUNT = 63480, entres.Conut = 126960. This code throws System.OutOfMemoryException. I tried to change architecture to x64 but it still throws the same exception.

How do I can fix this? I will be very grateful if you help me to solve this problem!

Your algorithm will allocate `63,480 * 126,960` (that's `8,059,420,800` instances) of `double` (8 bytes) values, that's `64,475,366,400` bytes, or **60 gigabytes**. No wonder you're running out of memory. — Dai, Dec 11 '16 at 09:53
The question is, what are you doing to do with this list. You should work not with lists but with enumerable and producer/consumer design pattern, so that garbage collector could eliminate not using data. — Maksim Simkin, Dec 11 '16 at 09:55
Given that the CLR, even in 64-bit mode, has a maximum object size of 2 gigabytes ( http://stackoverflow.com/questions/2338778/what-is-the-maximum-length-of-an-array-in-net-on-64-bit-windows ) it is unlikely you will solve your program with your current approach. You should think about how you can break your problem down into smaller parts and using hybrid disk/memory representations of your memory. Very few computers today have 128GB of RAM - as 64GB is insufficient to represent your program in practice - so you should rethink this. — Dai, Dec 11 '16 at 09:56

Dai · Accepted Answer · 2016-12-11T09:56:55.893

2

Disregarding the fact your program needs over 100GB of RAM to operate, if you know the size of a list beforehand then either preallocate it or use a fixed-size array: this avoids dynamic resizing and reallocations:

List<double> weights = new List<double>( count );
for( int j = 0; j < count; j++ )
{
     weights.Add( rand.NextDouble() );
}

or:

double[] weights = new double[count];
for( int j = 0; j < count; j++ )
{
     weights[j] = rand.NextDouble();
}

edited Dec 11 '16 at 09:56

answered Dec 11 '16 at 09:50

Dai

141,631
28
261
374

Why is this the approved answer? how will it solve the problem? it's like "disregarding the real problem this optimisation will not change anything", Your comment on the question should be marked as the answer – barakcaf Dec 12 '16 at 08:22
@barakcaf My comment of "rethink the problem" is not really an answer because it doesn't offer a concrete solution. At least for other people with similar problems there's a possibility that avoiding list reallocation will solve it for them. – Dai Dec 12 '16 at 09:25

score 0 · Answer 2 · answered Dec 11 '16 at 10:06

The .Net garbagge collector does not compacts large objects to avoid performace impact. Thus, you have 2 options:

Allocate once the array for large data.
Periodically set the value of the property GCSettings.LargeObjectHeapCompactionMode to GCLargeObjectHeapCompactionMode.CompactOnce. The next GC invocation will deal with the large objects, and the will reset to default value. See https://msdn.microsoft.com/en-us/library/system.runtime.gcsettings.largeobjectheapcompactionmode(v=vs.110).aspx

System.OutOfMemoryException while working with large Lists

2 Answers2