4

I am writing a program that needs to traverse a large 40gb binary file, but I only have 16gb of physical RAM. A friend told me that I can use file mapping to allievate this problem. I understand how to create a file mapping and reading into a file map handle, and how file mapping maps parts of a file in persistent memory to different chunks of virtual memory for reading.

So if I am understanding this correctly, I can create a buffer of say 10gb, and read the first 10gb of the file into this buffer. But When I have to read past the 10gb mark on the file, will the OS fetch another block automatically for me, or do I have to manually do so in my code?

riQQ
  • 9,878
  • 7
  • 49
  • 66
DebareDaDauntless
  • 471
  • 2
  • 7
  • 12

1 Answers1

6

The functions you linked to aren't (directly) related to file mapping. They're used for regular file I/O.

To use traditional file I/O with a really large file, you could do as you described. You would open the file, create a buffer, and read a chunk of the file into the buffer. When you need to access a different part of the file, you read a different chunk into the buffer.

To use a file mapping, you use CreateFile, CreateFileMapping, and then MapViewOfFile. You don't (directly) create a buffer and read a part of the file into it. Instead, you tell the system that you want to access a range of the file as though it were a range of memory addresses. Reads and writes to those addresses are turned into file i/o operations behind the scenes. In this approach you might still have to work in chunks. If the part of the file you need to access is not in the range you currently have mapped, you can create another view (and possibly close the other one).

But note that I said address space, which is different than RAM. If you're building for 64-bit Windows, you can try to map the entire 40 GB file into your address space. The fact that you have only 16 GB of RAM won't stop you. There may be some other problems at that size, but it won't be because of your RAM. If there are other problems, you'll be back to managing the file in chunks as before.

Adrian McCarthy
  • 45,555
  • 16
  • 123
  • 175
  • Ok. I am building a 64-bit program. So if I am understanding correctly I can use MapViewOfFile across the whole 40gb space, and the OS will handle the reads and writes? – DebareDaDauntless Jul 10 '15 at 20:44
  • In theory, yes. I can't guarantee that MapViewOfFile will succeed with a request that big, but it's worth a shot. I'm working on the assumption that you're reading an existing large file. If you're trying to create one, then you may be limited by the size of the paging file. – Adrian McCarthy Jul 10 '15 at 21:04