7

I am writing a c++ program to read/write a large file (probably larger than 60GB). By googling the problem, it seems that there is a 2GB limit on file io in 32 bit system (I am using windows 7 64bit but my program was compiled with mingw32). In my program, I am writing 10 integers at a time to the file and all these numbers are generated randomly based on some algorithm. It seems that the program can run even when the file size bigger than 40GB but there is no way for me to check if the data read by the program is really the one stored in the file or some junk numbers. But anyway, the program doesn't report any warning or error. Is this really possible to read/write file larger than 60GB in a 32-bit program?

Windows programmer
  • 7,871
  • 1
  • 22
  • 23
user1285419
  • 2,183
  • 7
  • 48
  • 70
  • Possible duplicate of http://stackoverflow.com/questions/301995/write-large-file – Carey Gregory Apr 08 '12 at 22:18
  • I think the link given by Carey Gregory answers the 2GByte file limit question. Why is there no way for you to check that the numbers are written correctly? Instead of writing random numbers, you could test by writing the sequence of numbers 0 to 2,147,483,647 (all positive 32 bit integers), which will be an 8GB file, then read it back. – gbulmer Apr 08 '12 at 23:36

1 Answers1

1

There's a limit on file size (4GB max, I think) on Fat32 file system. Windows 7 definitely shouldn't be using that filesystem by default.

Also on 32bit system there's a limit on the file size you can map into memory at once using CreateFileMapping/MapViewOfFile. However, fstream doesn't use CreateFileMapping/MapViewOfFile internally, so there's no limit for file size (aside from filesystem limits). And even with CreateFileMapping you can map portion of larger file into memory, so there's no limit aside from the one imposed by filesystem.

SigTerm
  • 26,089
  • 6
  • 66
  • 115
  • oh, if fstream doesn't really have limit on file size than it will be a great news for me since I have no idea how to break my file into pieces to avoid the 4GB limit. Thanks – user1285419 Apr 09 '12 at 00:49
  • 1
    "Windows 7 definitely shouldn't be using that filesystem by default." -- That statement is too broad. However, since the question involves a 40GB or 60GB file, it's probably not on USB flash memory. – Windows programmer Apr 09 '12 at 00:54