5

I'm working with large, and growing files using the managed wrappers for memory-mapped files: MemoryMappedFile, MemoryMappedViewAccessor.

I create empty files using this code:

long length = 1024L * 1024L * 1L; // 1MB

// create blank file of desired size (nice and quick!)
FileStream fs = new FileStream(filename, FileMode.CreateNew);
fs.Seek(length, SeekOrigin.Begin);
fs.WriteByte(0);
fs.Close();

// open MMF and view accessor for whole file
this._mmf = MemoryMappedFile.CreateFromFile(filename, FileMode.Open);
this._view = this._mmf.CreateViewAccessor(0, 0, MemoryMappedFileAccess.ReadWrite);

That works fine, up to 1GB. When I try 2GB, I get an IOException:

Not enough storage is available to process this command.

at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.MemoryMappedFiles.MemoryMappedView.CreateView(SafeMemoryMappedFileHandle memMappedFileHandle, MemoryMappedFileAccess access, Int64 offset, Int64 size)
at System.IO.MemoryMappedFiles.MemoryMappedFile.CreateViewAccessor(Int64 offset, Int64 size, MemoryMappedFileAccess access)
at (my code here)

I have a 64-bit version of Windows 7, the app is running as 64-bit, I have 6GB of RAM. All of that should be irrelevant, as far as I can tell though. These are large amounts of data, yes, however as I understand it the MemoryMappedFile and associated classes are the way to deal with large amounts of data like this.

As per the documentation, http://msdn.microsoft.com/en-us/library/dd267577.aspx, IOException literally means, 'An I/O error occurred'. However, the file is on disk just fine.

The app regularly increases file size as needed as mentioned, and in fact the error occurs at some point fairly randomly between ~400MB and ~2GB. When starting with 1GB, that always succeeds. When starting at the default 1MB, it fails much sooner, presumably due to releasing and re-allocating resources. (I always Flush and Close on the view, MMF and and streams).

I need to randomly-access the whole range of data. I am hoping that I don't need to dynamically maintain a dictionary of MemoryMappedViewAccessor objects - my interpretation on the virtual memory system used here would suggest that pages from a file of any size would be paged in and out as necessary by the memory system in Windows.

In the form of a question: why is this happening? How can I stop it? Is there a better way to achieve full, random, read-write access to files of any size? Up to 100GB for instance?

Kieren Johnstone
  • 41,277
  • 16
  • 94
  • 144
  • Do you have enough room on the drive you're writing the file to? MMF's map the file contents efficiently into memory, but still need to be backed by file storage. – Eric J. Sep 04 '11 at 15:55
  • Hundreds of gigs free, and if you look more closely at my explanation, the IOException does not occur before I'm creating the MMF or view accessor, and I create the file in the desired size just before that without error. Have resolved this, see answer below. :) – Kieren Johnstone Sep 04 '11 at 15:56
  • 4
    Everything points to your app actually running in 32-bit mode. Check the Platform target setting. – Hans Passant Sep 04 '11 at 15:57
  • @Hans - yes indeed, see my own answer from 3 mins ago ;) I was checking the wrong build config. – Kieren Johnstone Sep 04 '11 at 15:58
  • Grats on the upvote for something I already pointed out, by the way.. O_O – Kieren Johnstone Sep 04 '11 at 15:59
  • I believe that for very large files, you have no other option but to open chunks of the file at a time. There is no magical way to handle very large files... – Eli Iser Sep 04 '11 at 16:01
  • @Eli - well, you can use MMF for this purpose, until you run out of address space for your process. So limited by some fraction of 4GB in 32-bit mode, or some fraction of 1717986TB in 64-bit mode :) – Kieren Johnstone Sep 04 '11 at 16:03
  • @Keiren - hmm, I guess you're right. I was always under the impression the OS will try and handicap you somehow (like with the 2GB user-space addressing in 32 bit). – Eli Iser Sep 04 '11 at 16:06

1 Answers1

4

The app was indeed set to target x86 rather than x64 in the specific project build configuration I had selected.

My guess is my process address space had become full, since it was running in 32-bit mode.

Solution - change platform target to x64 and run on a 64-bit OS.

Kieren Johnstone
  • 41,277
  • 16
  • 94
  • 144
  • @Eric actually 32bit processes can access 4GB when on a 64bit OS. – Tim Lloyd Sep 04 '11 at 16:06
  • 1
    To be really pedantic, sometimes you can get 3GB in 32-bit on Windows. As usual Raymond Chen covers [it](http://blogs.msdn.com/b/oldnewthing/archive/2004/08/22/218527.aspx). – user786653 Sep 04 '11 at 16:07
  • .NET 4.0 Client Profile (the default for C# console apps) strikes again! – Jonathan Oliver Jul 13 '12 at 11:58
  • @Jonathan Oliver: Que? Sure you mean Client **Profile** and not x86/x64 (**Platform**) ? – Ruben Bartelink Jul 13 '12 at 21:05
  • Client Profile defaults to x86. – Jonathan Oliver Jul 13 '12 at 21:42
  • Hmm, wierd. [The MSDN high level doc for Client Profile](http://msdn.microsoft.com/en-us/library/cc656912.aspx) doesnt mention it except to list project types that are CP by default. The debate on [this Connect issue](https://connect.microsoft.com/VisualStudio/feedback/details/455333/platform-target-is-defaulting-to-x86-rather-than-any-cpu). There is [a VS2010 era rationale for x86](http://blogs.msdn.com/b/rmbyers/archive/2009/06/8/anycpu-exes-are-usually-more-trouble-then-they-re-worth.aspx) and in VS2012 that matures to [`AnyCpuPrefer32Bit`](http://stackoverflow.com/a/11037492/11635) – Ruben Bartelink Jul 14 '12 at 00:31
  • IOW AFAIUI everything I know says the defaulting to CP and defaulting to x86 in VS2010 are are orthogonal concerns. Even the AnyCpuPrefer32Bit stuff is just a 'and do the Right Thing on ARM to'. But I know you understand all this perfectly well (thanks for your fantastic EventStore BTW), so are you saying changing the target dropdown to CP can also force a change from x64/anyCpu to x86 at the same time - or have you got a link that provides somme more detail? – Ruben Bartelink Jul 14 '12 at 00:36
  • Mmf should not map such huge size, if you access mmf via chunks of smaller size, it will perform better. – Akash Kava Dec 02 '13 at 06:12