1

I'm using some very large collections (lists and dictionaries), containing more than Int32.MaxValue items.
The machines running my application may have anywhere from 64 - 1024 GB of RAM, so memory space is not a concern at the moment.

I need to know two things:

a) Can the System.Collections.Generic objects handle 64-bit item counts?

b) If so, is there a way to set the initial capacity to a value greater than Int32.MaxValue?

I've been reading about the significant performance benefits that come with setting the initial capacity exactly where you need it - e.g. here.
I imagine the importance of setting the initial capacity only becomes more and more significant as collections get larger and larger.

Giffyguy
  • 20,378
  • 34
  • 97
  • 168
  • 1
    Possible duplicate of [List size limitation in C#](http://stackoverflow.com/questions/7885294/list-size-limitation-in-c-sharp) – Fenton Oct 21 '15 at 19:53
  • 1
    That's a really large collection. Is there really no better approach than having more than some 2 billion elements in memory? – ryanyuyu Oct 21 '15 at 19:54
  • 1
    I'm sorry then. Good luck. For what it's worth, Steve's duplicate target is a good place to start. – ryanyuyu Oct 21 '15 at 19:55
  • 6
    AFAIK arrays are limited to about 2 billion entries, even on 64 bit. By default they're even limited to 2 GiB, but that limitation can be disabled via the configuration file. So you'll need to use custom collections, backed by several arrays if you want them to be that big. – CodesInChaos Oct 21 '15 at 19:56
  • 2
    Building your own collection is the way to go, at these sizes that's the most sensible thing to do: build it to match your requirements. – Lucas Trzesniewski Oct 21 '15 at 19:57
  • 1
    yes that why you can get long Count of list. there is property `LongCount`. also if there is no memory concern then use two lists. 4 lists etc.... – M.kazem Akhgary Oct 21 '15 at 20:03
  • 2
    @M.kazemAkhgary AFAIK `List` and `IList` have no `LongCount` can't be that big in the current implementation. – CodesInChaos Oct 21 '15 at 20:12
  • 1
    @CodesInChaos oops. thats from linq. any way im sure there is `LongLength` property for arrays. – M.kazem Akhgary Oct 21 '15 at 20:18
  • 1
    @M.kazemAkhgary The array API has been designed to support arrays with more than 2^31 items, but the current implementation of the runtime doesn't support them. – CodesInChaos Oct 21 '15 at 20:43
  • Have you actually calculated how much memory your objects will consume? Hint: 2GB of 64bit indexes are 17GB alone. – Erik Funkenbusch Oct 21 '15 at 21:00
  • @ErikFunkenbusch Hint: I already addressed this in my question. However, since you are asking, this is a very detail-oriented performance-driven application design - yes, we have calculated how much memory will be consumed, this is simply not relevant to the specific question at hand. Also, people said the same thing decades ago, when people started asking about arrays with greater than 2^15 items ... same story, people just needed to acquire a machine that could handle the load. – Giffyguy Oct 21 '15 at 21:13
  • @Giffyguy - I'm more than aware you can acquire a machine to handle the load, I questioned whether or not the machine you specified would handle the load, which I don't think It will. As I said, just the pointer array alone will be 17GB, if your objects have any size to them you'll top out 1024GB in a flash. – Erik Funkenbusch Oct 22 '15 at 01:55

1 Answers1

1

According to the MSDN docs on Lists and HashSets, you can set the maximum capacity to 2 billion on 64 bit machines.

For very large List objects, you can increase the maximum capacity to 2 billion elements on a 64-bit system by setting the enabled attribute of the configuration element to true in the run-time environment.

Necoras
  • 6,743
  • 3
  • 24
  • 45
  • I'm having trouble finding the section where your quote comes from. Can you provide a direct link to the page? Ctrl-F couldn't find it on the general List page, and Google can't find it either. – Giffyguy Oct 21 '15 at 20:13
  • 1
    @Giffyguy It's down in the "Remarks" section. Search for "billion" and it should take you right to it. – Necoras Oct 21 '15 at 20:20
  • 1
    Thanks. Any chance you know which configuration element they are referring to? – Giffyguy Oct 21 '15 at 20:39
  • 3
    @Giffyguy [`gcAllowVeryLargeObjects`](https://msdn.microsoft.com/en-us/library/hh285054.aspx) but this only removes the 2 GB limit, not the 2.1 billion (slightly below 2^31) element limitation. – CodesInChaos Oct 21 '15 at 20:45
  • @CodesInChaos Oh good, so it is referring to the same config option that the other question mentioned. Thanks for confirming. – Giffyguy Oct 21 '15 at 20:55