0

I get an out of memory exception when running this line of code:

double[,] _DataMatrix = new double[_total_traces, _samples_per_trace];

But this code completes successfully:

double[][] _DataMatrix = new double[_total_traces][];
for (int i = 0; i < _total_traces; i++)
{
    _DataMatrix[i] = new double[_samples_per_trace];
}

My first question is why is this happening?

As a followup question, my ultimate goal is to run Principal Component Analysis (PCA) on this data. It's a pretty large dataset. The number of "rows" in the matrix could be a couple million. The number of "columns" will be around 50. I found a PCA library in the Accord.net framework that seems popular. It takes a jagged array as input (which I can successfully create and populate with data), but I run out of memory when I pass it to PCA - I guess because it is passing by value and creating a copy of the data(?). My next thought was to just write my own method to do the PCA so I wouldn't have to copy the data, but I haven't got that far yet. I haven't really had to deal with memory management much before, so I'm open to tips.

Edit: This is not a duplicate of the topic linked below, because that link did not explain how the memory of the two was stored differently and why one would cause memory issues despite them both being the same size.

Zan Sullivan-Wilson
  • 1
  • 1
  • Possible duplicate of [What are the differences between a multidimensional array and an array of arrays in C#?](https://stackoverflow.com/questions/597720/what-are-the-differences-between-a-multidimensional-array-and-an-array-of-arrays) – mjwills May 15 '18 at 14:24

1 Answers1

1

In 32bits it is complex to have a continuous range of addresses of more than some hundred mb (see for example https://stackoverflow.com/a/30035977/613130). But it is easy to have scattered pieces of memory totalling some hundred mb (or even 1gb)...

The multidimensional array is a single slab of continuous memory, the jagged array is a collection of small arrays (so of small pieces of memory).

Note that in 64bits it is much easier to create an array of the maximum size permitted by .NET (around 2gb or even more... see https://stackoverflow.com/a/2338797/613130)

xanatos
  • 109,618
  • 12
  • 197
  • 280
  • Thank you! Your explanation is very helpful. I changed the configuration to x64 and I could declare the multidimensional array successfully without running out of memory. – Zan Sullivan-Wilson May 15 '18 at 14:49
  • https://learn.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/runtime/gcallowverylargeobjects-element may also be of interest @ZanSullivan-Wilson. – mjwills May 15 '18 at 23:11