7

It seems there is a 2 GB size limit for objects in .NET: How to run Fsi.exe in 64 Bits?

Is there a work around this? I would like to load a very large float array (10 GB) in memory and then do some work.

Wai Ha Lee
  • 8,598
  • 83
  • 57
  • 92
jlezard
  • 1,417
  • 2
  • 15
  • 32

3 Answers3

13

.NET limits any object to max 2 GB even on 64 bit platforms. You can create your own data type, that uses multiple objects to store more data, thus getting around the 2 GB limit of a single object. For instance a List<float[]> would allow you to store more than 2 GB, but you would have to write the necessary plumbing code to make it behave similar to a single, large array.

You may also want to check this question.

Community
  • 1
  • 1
Brian Rasmussen
  • 114,645
  • 34
  • 221
  • 317
  • Thanks. Do you know what are the limitations in Java and Python for example ? – jlezard Nov 30 '10 at 09:45
  • @jlezard: Sorry, I can't help you with the specifics for Java and Python. IronPython would have the same limit as it runs on .NET but apart from that I can't say. – Brian Rasmussen Nov 30 '10 at 11:35
5

In versions of .NET prior to 4.5, the maximum object size is 2GB. From 4.5 onwards you can allocate larger objects if gcAllowVeryLargeObjects is enabled. Note that the limit for string is not affected, but "arrays" should cover "lists" too, since lists are backed by arrays.

Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
2

I don't think there is an easy workaround for this, it seems to me there could be difficulties implementing a heap without a 2Gb limit on object size.

Maybe you would be better breaking the data up some how. It should be possible to write a class that behave like an array but splits the data up into fixed sized chunks under the hood.

Andy Lowry
  • 1,767
  • 13
  • 12
  • With .Net 4.5 this 2 GB limit is not in the picture anymorehttps://bhrnjica.net/2012/07/22/with-net-4-5-10-years-memory-limit-of-2-gb-is-over/ – RBT Aug 16 '16 at 23:39