My structure array runs out of memory in C# how do I fix it?

Question

Hello I must store all the elements of this structure in memory for processing, I cannot just load chunks of it at a time I must have it all loaded at any point so I'm trying to fit it all in the memory somehow it runs out of memory even though I have about 20 GB of RAM and maybe 17 GB left.. I understand that Collisions have a internal limit of 2 GB so RAM doesn't matter in this sense at all.

Here is how my layout looks like

public struct REGTYPE
{
    public byte REG_Kind; // ;1=8 bits \ 2=16 bits \ 3=32 bits \ 4=MMX \ 5=XMM \ 6=Float stack \ 7=Segment \ 8=Debug \ 9=Control \ 10=Test
    public byte REG_Ptr_Kind; // ;1=Byte PTR \ 2=Word PTR \ 3=Dword PTR \ 4=Qword PTR \ 5=mmword ptr \ 6=xmmword ptr \ 7=FWord PTR \ 8=tbyte ptr \ 9=null ptr (LEA)
    public byte REG_Type; //  ;0-7= direct register index \ 16 register=byte && 7 \ 32 register=(byte && 63)/8 \ 64=[32/16 address only] \ 128=[using x86 relatives]
    public byte REG_BaseAsReg; // ? ;1=Register only (BASE exposed)!
}

public struct REGSTRUCT
{
    public uint SEG_TYPE;
    public uint Base;
    public uint INDEX;
    public uint SCALE;
    public uint DISPLACEMENTS;
    public uint DISPLACEMENT_TYPE;
    public REGTYPE REG_Kind;
    public uint PTR_TYPE;
}

public struct IMMSTRUCT
{
    public uint VALUE_LO;
    public uint VALUE_HI;
    public uint VALUE_TYPE; //     1=Byte \ 2=Word \ 4=Dword \ 8=ByteToWord \ 16=ByteToDword \ 32=AbsJump \ 64=ShortJump \ 128=LongJump
}


public struct DisAsmStruct
{
    public uint Instruction_Prefix;
    public uint Instruction;
    public REGSTRUCT Reg1;
    public REGSTRUCT Reg2;
    public uint Reg_Reg; //1=from ptr
    public IMMSTRUCT Imm;
    public uint Instruction_Length;
}

public struct AsmStruct
{
    public uint Address;
    public string ASM;
    public DisAsmStruct disASM; //tried to add ref keyword no success
}

public static AsmStruct[] AsmCache = null;

In the middle of running this code it crashes on the Array.Resize line stating it ran out of memory when AshCache reaches about 2 million elements it has to go up to 10 maybe 20 million elements before it's done loading, I know it's possible to pack all this in structs without running out of memory as I did it in C++, I also believe I know the cause of this problem, the cause is that it stores the Structure as a copy of itself instead of just a reference to that struct, so twice as much memory is wasted for every struct how do I store a struct by reference in AsmCache? The reason I use Array.Resize is to increase the collection limit by 1000 everytime more instructions are found.. I don't just set to limit to some arbitrary limit. This is how I used to do it in Vb6 (ReDim Preserve)

DisAsmStruct DisA = new DisAsmStruct();
AsmCache = new AsmStruct[1001];
Array.Resize(ref AsmCache, AsmCache.GetUpperBound(0) + 1001);
AsmCache[Number + 1].Address = BaseAddress + CNT;
AsmCache[Number + 1].disASM = ref DisA; //this line will error as I tried to fix it (remove ref to work)
AsmCache[Number + 1].ASM = OpCode; //Like "MOV EAX, [ECX+34h]" blah blah

Answer 1

If you want to allow objects that are larger than 2GB you will need to allow them. Below would go into your app.config file or another config file depending on your project type.

<configuration>
    <runtime>
        <gcAllowVeryLargeObjects enabled="true" />
    </runtime>
</configuration>

Documentation

https://msdn.microsoft.com/en-us/library/hh285054(v=vs.110).aspx

Answer 2

Make AsmStruct a class, otherwise when You resize the array (or the List resizes itself) You copy all that data into new location (which is very slow btw). With AsmStruct as a class You copy just pointers. I made a simulation of Your resizing strategy and with class You can create 4x more items before OutOfMemory.

With editbin trick .NET Out Of Memory Exception - Used 1.3GB but have 16GB installed You can push the limit to 2x more.

If this is not enough and You have only 32bit library, You would have to split the program somehow. One 32bit program to communicate with library and one 64bit for processing.