System.OutOfMemoryException. Creating a big matrix

Question

I have a matrix [3,15000]. I need to count covariance matrix for the original matrix and then find its eigenvalues.

This is a part of my code:

double[,] covarianceMatrix = new double[numberOfObjects,numberOfObjects];
for (int n=0; n<numberOfObjects;n++)
    {
    for (int m=0;m<numberOfObjects;m++)
    {
        double sum = 0;
        for (int k=0; k<TimeAndRepeats[i,1]; k++)
        {
            sum += originalMatrix[k,n]*originalMatrix[k,m];
        }
    covarianceMatrix[n,m] = sum/TimeAndRepeats[i,1];
    }
}
alglib.smatrixevd(covarianceMatrix,numberOfObjects,1,true,out eigenValues, out eigenVectors);

NumberOfObjects here is about 15000. When I do my computations for a smaller number of objects everything is Ok, but for all my data I get an exeption. Is it possible to solve this problem?

I am using macOS, x64

My environment is MonoDevelop

score 4 · Accepted Answer · edited May 23 '17 at 10:25

4

double[,] covarianceMatrix = new double[numberOfObjects,numberOfObjects];

You said that your matrix is [3, 15000] and that numberOfObjects is 15000. By this line of code here, you're creating a matrix of [15000, 15000] of doubles

15000 * 15000 = 225000000 doubles at 8 bytes each: 1,800,000,000 bytes or 1.8GB

That's probably why you are running out of memory.

Edit:

According to this question and this question the size of objects in C# cannot be larger that 2GB. The 1.8GB does not count any additional overhead required to reference the items in the array, so that 1.8GB might actually be > 2GB when everything is accounted for (Can't say without the debugging info, someone with more C# experience might have to set me straight on this). You might consider this workaround if you're trying to work with really large array, since statically allocated arrays can get messy.

edited May 23 '17 at 10:25

Community

1
1

answered Jul 16 '13 at 15:02

Justin Pearce

4,994
2
24
37

CanI somehow allow my program use more memory to avoid this exception? – user2080209 Jul 16 '13 at 15:17
Doesn't x64 support addressing much more memory per process? Or maybe it's a Mac thing.... this wouldn't throw on x64 in windows. – Vivek Jul 16 '13 at 18:04
1

@Vivek x64 allows for larger integer values and memory addresses (since you're talking about 64 1's and 0's as opposed to 32. It does not change the max sizes for data structures handled by C# – Justin Pearce Jul 16 '13 at 19:22

score 0 · Answer 2 · answered Jul 16 '13 at 15:02

0

When you create covarianceMatrix, you are creatinf an object of 15000*15000 = 225000000

so you need 1800000000 bytes of memory. it is because of that that you have OutofMemoryException

answered Jul 16 '13 at 15:02

Swift

1,861
14
17

Is it something about settings in monodevelop? – user2080209 Jul 16 '13 at 15:12
1800000000 it about 1.7 GB, it means you must have 1.7 GB of free RAM that can be allocated to your program (well I am not getting in detail, and I am simplifying the think here), so I think when you are in more powerful machine you will be able to do this calculation without problem. – Swift Jul 16 '13 at 19:13

score 0 · Answer 3 · answered Jul 16 '13 at 15:38

Exception name tells you exactly what's the problem. You could use floats instead of doubles to bisect ammount of memory needed. Other option would be to create some class object for a covariance matrix that would save data in a disk file, though you'd need to implement proper mechanisms to operate on it and the performance would be limited aswell.

System.OutOfMemoryException. Creating a big matrix

3 Answers3