2

In a Delphi program, I have a dynamic array with 4,000,000,001 cardinals. I'm trying to write (and later read) it do a drive. I used the following:

const Billion = 1000000000;

stream := tFileStream.Create( 'f:\data\BigList.data', fmCreate);
stream.WriteBuffer( Pointer( BigArray)^, (4 * billion + 1) * SizeOf( cardinal));
stream.free;

It bombed out with: ...raised exception class EWriteError with message 'Stream write error'.

The size of the file it wrote is only 3,042,089KB.

Am I doing something wrong? Is there a limit to the size that can be written at once (about 3GB)?

Jud McCranie
  • 58
  • 9
  • 2
    Dumb questions: 1. Did you have enough disk space (4bn * cardinals will require 16Gb). 2. What's the file system of the disk you are writing to ? Some file systems have file size limits (e.g. 4Gb per single file on FAT32). 3. If F: is a network drive, have you tried writing the file in smaller increments ? 4. Are you compiling to 32-bit or 64-bit ? If 32-bit then you may be running into a problem addressing your BigArray (though the ~3Gb limit you appear to be reaching is a little unexpected, if that were the case). – Deltics Jul 04 '17 at 23:44
  • 1
    Why are you calculating the number of bytes this way? You could just ask the array: `stream.WriteBuffer( Pointer(BigArray)^, Length(BigArray) * SizeOf(Cardinal));` on the other hand, the length of a dynamic array is expressed as an integer, so it can't even hold 4 billion cardinals to begin with. – Remy Lebeau Jul 05 '17 at 02:44
  • 1. Yes, the drive has about 2TB free. 2. NTFS. 3. F is an internal HD. 4. 64-bit platform. – Jud McCranie Jul 05 '17 at 04:02
  • 1. Yes, the drive has about 2TB free. 2. NTFS. 3. F is an internal HD. 4. 64-bit platform. ------------------ I'm calculating the size this way because (even though it is a dynamic array), it has a fixed size (and larger than a static array can be). If a dynamic array can hold only 2^31 elements (I wish they would make Delphi fully 64-bit), then that could cause a problem. I checked the 3 billionth term and the 4 billionth term of the array, but it may be wrapping around without me knowing it. – Jud McCranie Jul 05 '17 at 04:08
  • var cc : cardinal; BigArray : array of cardinal; SetLength( BigArray, 4 * billion + 1); for cc := 1 to 4 * billion do BigArray[ cc] := cc; cc := 200 * millionInt64; repeat writeln( cc:12, BigArray[ cc]:12); inc( cc, 200 * MillionInt64); until cc > 4 * billion; gives correct output. 200000000 200000000 ... 3800000000 3800000000 4000000000 4000000000 – Jud McCranie Jul 05 '17 at 04:20
  • Is there way to get a CR/LF in these messages? – Jud McCranie Jul 05 '17 at 04:21
  • Dynamic arrays can be sized to 2^63 in the 64 bit compiler, that's not your problem. – David Heffernan Jul 05 '17 at 04:37

1 Answers1

5

The Count parameter of WriteBuffer is a 32 bit integer so you cannot pass the required value in that parameter. You will need to write the file with multiple separate calls to WriteBuffer, where each call passes a count that does not exceed this limit.

I suggest that you write it something like this. 

var
  Count, Index, N: Int64;
.... 
Count := Length(BigArray);
Index := 0;
while Count > 0 do begin
  N := Min(Count, 8192);
  stream.WriteBuffer(BigArray[Index], N*SizeOf(BigArray[0]));
  inc(Index, N);
  dec(Count, N);
end;

An additional benefit is that you can readily display progress.

David Heffernan
  • 601,492
  • 42
  • 1,072
  • 1,490