Resizing big arrays

Question

I have an array which size is like 2 GB (filled with audio samples). Now I want to apply a filter for that array. This filter is generating like 50% more samples than input source. So now I need to create new array which size is 3 GB. Now I gave 5 GB of memory used. But if this filter can operate only at that source array and only need some more space in this array. Question: can I allocate a memory in C# that can be resized w/o creating a second memory block, then removing that first one? I just thought, If memory in PC's is divided into 4 kB pages (or more), so why C# cannot (?) use that good feature?

Answer 1

Question: can I allocate a memory in C# that can be resized w/o creating a second memory block, then removing that first one?

No, you cannot resize an array in .NET. If you want to increase the size of an array you will have to create a new and bigger array and copy all the data from the existing array to the new array.

To get around this problem you could provide your own "array" implementation based on allocating smaller chunks of memory but presenting it as one big buffer of data. An example of this is StringBuilder that is based on an implementation of chunks of characters, each chunk being a separate Char[] array.

Another option is to use P/Invoke to get access to low level memory management functions like VirtualAlloc that allows you to reserve pages of memory in advance. You need to do this in a 64 bit process because the virtual address space of a 32 bit process is only 4 GB. You probably also need to work with unsafe code and pointers.

Answer 2

If your filter can work in-place just allocate 50% more space at the beginning. All you need to know is the actual length of the original sample.

If that code doesn't work always and you don't want to consume more memory beforehand, you can allocate half of the original array (the extension array) and check which part your access relates to:

byte[] myOriginalArray = new byte[2GB]; // previously allocated 

byte[] myExtensionArray = new byte[1GB]; // 50% of the original
for(... my processing code of the array ...)
{
  byte value = read(index);
  ... process the index and the value here
  store(index, value);
}

byte read(int index)
{
  if(index < 2GB) return myOriginalArray[index];
  return myExtensionArray[index - 2GB];
}

void store(int index, byte value)
{
   if(index < 2GB) myOriginalArray[index] = value;
   myExtensionArray[index - 2GB] = value;
}

You add index check and subtraction overhead for each access to the array. That could also be made smarter for certain cases. For instance for the portion you do not need to access extension you can use your faster loop and for the part where you need to write to extension part you can use the slower version (two consecutive loops).