1

I need to work with a large amount of data in memory. I am loading it from an SQLite database on SSD and using EF6 to construct business objects from it. As soon as the Process Memory window shows usage hitting 3.2GB I get an Out Of Memory exception from Entity Framework.

For now I am just loading into lists. I had read somewhere that there were limits on the sizes of list structures so instead of using one big list I have created multiple simple DataBlock container objects to each hold a chunk of the required data. It doesnt seem to make any difference. The PC has plenty of RAM (16GB). I am using a new context to populate each DataBlock and then destroying them.

For Each DataBlock In DataBlocks
    Using Context As New mainEntities
        Dim FirstRecordTimeUTC As Long = TimeFunctions.ConvertDateToUTC(DataBlock.StartDate)
        Dim LastRecordTimeUTC As Long = TimeFunctions.ConvertDateToUTC(DataBlock.EndDate)

        Dim CandlesInRange = (From Cand In Context.HistoricalRecords
                              Where Cand.time_close_utc >= FirstRecordTimeUTC
                              Where Cand.time_close_utc <= LastRecordTimeUTC
                              Order By Cand.id
                              Select Cand).ToList
        DataBlock.AllCandles = CandlesInRange

        Dim RTsInRange = (From Cand In Context.HistoricalRecordRealTimes
                          Where Cand.time_close_utc >= FirstRecordTimeUTC
                          Where Cand.time_close_utc <= LastRecordTimeUTC
                          Order By Cand.id
                          Select Cand).ToList
        DataBlock.AllRTs = RTsInRange

        Dim StatsInRange = (From Cand In Context.InstrumentStats
                            Where Cand.time_close_utc >= FirstRecordTimeUTC
                            Where Cand.time_close_utc <= LastRecordTimeUTC
                            Order By Cand.id
                            Select Cand).ToList
        DataBlock.AllStats = StatsInRange
    End Using 
Next

The compiler platform is set to 'Any CPU'. System is as follows:

Win 10 64, VS 2017, 16GB RAM, Ryzen 5 3600

Any thoughts on what I am doing wrong would be much appreciated.

C Perkins
  • 3,733
  • 4
  • 23
  • 37
TripleAntigen
  • 2,221
  • 1
  • 31
  • 44
  • It's not about system memory specifically but process memory. Look into allocating more memory to your process. – jmcilhinney Nov 03 '19 at 08:03
  • 2
    Projects targeting AnyCPU and .NET 4.5 and higher by default use the configuration "AnyCPU 32-bit preferred". This means that your application will always be compiled as a 32-bit app (which are by default limited to 4 GB of RAM) even on a 64-bit system, the exception being when you're running on an ARM processor (in which case it will be compiled to ARM). – Visual Vincent Nov 03 '19 at 08:04
  • 2
    You need to target the classic AnyCPU configuration in order for your app to compile as a 64-bit app on your 64-bit system. Right-click your project in the Solution Explorer and press `Properties`, go to the `Compile` tab and uncheck the box labeled `Prefer 32-bit` (it might be under `Advanced Compile Options` as well... I don't have VS in front of me at the moment). – Visual Vincent Nov 03 '19 at 08:10
  • Depends largely on what you do with these lists. The preferred approach is to try and convert this process to a streaming solution. – Gert Arnold Nov 03 '19 at 09:21
  • 1
    Thank you so much Visual Vincent that did the trick! If you made it into an answer I would mark it as the solution. You're a legend. – TripleAntigen Nov 03 '19 at 09:55
  • 1
    Possible duplicate of [What is the purpose of the "Prefer 32-bit" setting in Visual Studio and how does it actually work?](https://stackoverflow.com/questions/12066638/what-is-the-purpose-of-the-prefer-32-bit-setting-in-visual-studio-and-how-does) – Visual Vincent Nov 03 '19 at 11:39
  • 2
    Glad I could help. I think the duplicate I suggested is better suited as an "answer" as it explains a bit more thorougly what `Prefer 32-bit` is and how it works. – Visual Vincent Nov 03 '19 at 11:40

1 Answers1

0

Visual Vincent has answered this question best in the comments:

Projects targeting AnyCPU and .NET 4.5 and higher by default use the configuration "AnyCPU 32-bit preferred". This means that your application will always be compiled as a 32-bit app (which are by default limited to 4 GB of RAM) even on a 64-bit system, the exception being when you're running on an ARM processor (in which case it will be compiled to ARM).

You need to target the classic AnyCPU configuration in order for your app to compile as a 64-bit app on your 64-bit system. Right-click your project in the Solution Explorer and press Properties, go to the Compile tab and uncheck the box labeled Prefer 32-bit (it might be under Advanced Compile Options as well... I don't have VS in front of me at the moment).

He also referenced another question that explains this setting: What is the purpose of the “Prefer 32-bit” setting in Visual Studio and how does it actually work?

TripleAntigen
  • 2,221
  • 1
  • 31
  • 44