how do you create a large array?

Question

Given

// r is a System.Data.IDataRecord
var blob = new byte[(r.GetBytes(0, 0, null, 0, int.MaxValue))];
r.GetBytes(0, 0, blob, 0, blob.Length);

and r.GetBytes(...) returns Int64 Since Array.zeroCreate and Array.init take Int32 how do I create an empty array that is potentially larger than Int32.MaxValue?

Are you sure you want to create an array that is potentially larger than 2GB? — Fyodor Soikin, Jun 06 '16 at 17:34
Agree with @FyodorSoikin - it's very unlikely that your PC / server can handle 2.1 billion+ contiguous memory addresses adequately. *What problem are you trying to solve?* — Haney, Jun 06 '16 at 17:35
I'm not sure if it is dense or sparse, I'm just trying to translate what C# is already doing into valid (maybe even idiomatic functional) F# where possible. — Maslow, Jun 06 '16 at 19:42
So does it even matter if in F# it is an array? It sounds like what you really want to know is how can I translate this C# to F# so that it works? — Guy Coder, Jun 06 '16 at 20:27
Of interest: [OCaml Bigarray](http://caml.inria.fr/pub/docs/manual-ocaml/libref/Bigarray.html) I know this is OCaml, but F# was derived from OCaml and someone may have translated this to F# in a third party library. — Guy Coder, Jun 06 '16 at 20:32
@GuyCoder F# uses `System.Array` everywhere. `[| 3 |]` is a `System.Array`. `Microsoft.FSharp.Collections.Array` is just a module containing functions that operate on standard .NET arrays; it is not a different array type. — Asik, Jun 06 '16 at 20:57
Of interest: [How to: Configure Projects to Target Platforms](https://msdn.microsoft.com/en-us/library/ms185328.aspx) — Guy Coder, Jun 06 '16 at 22:40

score 6 · Answer 1 · answered Jun 06 '16 at 20:25

In .NET 4.5 and newer, the gcAllowVeryLargeObjects configuration element allows you to create individual objects that are larger than 2GB, but even that does not change the hard limit on number of elements in an array. To quote from the Remarks section:

The maximum number of elements in an array is UInt32.MaxValue.

The maximum index in any single dimension is 2,147,483,591 (0x7FFFFFC7) for byte arrays and arrays of single-byte structures, and 2,146,435,071 (0X7FEFFFFF) for other types.

I'm not entirely sure why UInt32 rather than Int32 (you can create array where the smallest index is Int32.MinValue, but I'm not sure you'll be able to do much with it :-)).

However, with the gcAllowVeryLargeObjects parameter, you can create very large arrays by creating an array of structs that contain multiple values - this allocates continuous memory block over 2GB:

[<Struct>]
type Quadruple<'T>(v1:'T, v2:'T, v3:'T, v4:'T) = 
  member x.V1 = v1
  member x.V2 = v2
  member x.V3 = v3
  member x.V4 = v4

let qa : Quadruple<byte>[] = Array.zeroCreate 10

I don't think this is particularly practical, but it is one possibility. In reality, it seems easier just to split your data into an array of arrays. One level of indirection will probably not cause too much overhead, when the individual arrays are around 2GB.

Asik · Accepted Answer · 2016-06-06T20:22:35.860

3

You can't. The maximum length of a single-dimensional array on .NET is System.Int32.MaxValue. Your C# code raises an OverflowException when the value is greater than this limit. The equivalent F# code is:

let blob =         
    let length = r.GetBytes(0, 0, null, 0, Int32.MaxValue)
    if length > int64(Int32.MaxValue) then
        raise (OverflowException())
    else
        Array.zeroCreate<byte>(int32(length))

edited Jun 06 '16 at 20:22

answered Jun 06 '16 at 20:17

Asik

21,506
6
72
131

that's all the info I was looking for. (doing the exact same thing the C# code would do in F#) excellent. – Maslow Jun 07 '16 at 13:02

how do you create a large array?

2 Answers2