C# array to implement support for page-file architecture

Question

Let me explain what I need to accomplish. I need to load a file into RAM and analyze its structure. What I was doing is this:

//Stream streamFile;

byte[] bytesFileBuff = new byte[streamFile.Length];
if(streamFile.Read(bytesFileBuff, 0, streamFile.Length) == streamFile.Length)
{
    //Loaded OK, can now analyze 'bytesFileBuff'
    //Go through bytes in 'bytesFileBuff' array from 0 to `streamFile.Length`
}

But in my previous experience with Windows and 32-bit processes, it seems like even smaller amounts of RAM can be hard to allocate. (In that particular example I failed to allocate 512MB on a Windows 7 machine with 16GB of installed RAM.)

So I was curious, is there a special class that would allow me to work with the contents on a file of hypothetically any length (by implementing an internal analog of a page-file architecture)?

Does your analysis require random access to the file? or can you do it linearly? Even if you need multiple passes, with files that large you are always better off processing it as a stream. — Euro Micelli, Mar 25 '13 at 02:14
@EuroMicelli: Yeah, I'm leaning towards using streams too. It's just too complicated to work with them (or I should say, not as easy as with a byte array.) — c00000fd, Mar 25 '13 at 02:25

score 2 · Accepted Answer · answered Mar 25 '13 at 03:13

If linear stream access (even with multiple passes) is not a viable option, the solution in Win32 would be to use Memory Mapped Files with relatively small Views.

I didn't think you could do that in C# easily, but I was wrong. It turns out that .NET 4.0 and above provide classes wrapping the Memory Mapped Files API.

See http://msdn.microsoft.com/en-us/library/dd997372.aspx

If you have used memory mapped files in C/C++, you will know what to do.

The basic idea would be to use MemoryMappedFileCreateFromFile to obtain a MemoryMappedFile object. With the object, you can call the CreateViewAccessor method to get different MemoryMappedViewAccessor objects that represent chunks of the file; you can use these objects to read from the file in chunks of your choice. Make sure you dispose MemoryamappedViewAccessors diligently to release the memory buffer.

You have to work out the right strategy for using memory mapped files. You don't want to create too many small views or you will suffer a lot of overhead. Too few larger views and you will consume a lot of memory.

(As I said, I didn't know about these class wrappers in .NET. Do read the MSDN docs carefully: I might have easily missed something important in the few minutes I spent reviewing them)

C# array to implement support for page-file architecture

1 Answers1