0

Question: Is there a way to read files in excess of 2GB using MemoryMappedViewStream?

I am attempting to read log files that are anywhere from 1 to 12 GB. The 1GB files read OK, but I am receiving the following error when reading a 4GB file:

System.IO.IOException HResult=0x80070008 Message=Not enough storage is available to process this command.

Source=System.Core StackTrace: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.MemoryMappedFiles.MemoryMappedView.CreateView(SafeMemoryMappedFileHandle memMappedFileHandle, MemoryMappedFileAccess access, Int64 offset, Int64 size) at System.IO.MemoryMappedFiles.MemoryMappedFile.CreateViewStream(Int64 offset, Int64 size, MemoryMappedFileAccess access) at System.IO.MemoryMappedFiles.MemoryMappedFile.CreateViewStream() at ExchIISParserLib.LogParser.ParseLogs(Int32 daysago) in ...

My system has plenty of disk and memory space available to read the 4GB file. The line of code in question is:

MemoryMappedViewStream memoryMappedViewStream = MemoryMappedFile.CreateFromFile(log, FileMode.Open).CreateViewStream();

In my research efforts, I have found that MemoryMappedViewStream seems to be problematic when files exceed 2GB.

https://stackoverflow.com/a/49738895/4630376

I have looked at the offset and size parameters for the CreateViewStream() method. But those appear to just create a static window over the specified file, which does not read the entire file.

J Weezy
  • 3,507
  • 3
  • 32
  • 88
  • Probably a silly question but (as described in the link you posted), you aren't running a 32-bit process are you? – BernieP Dec 10 '18 at 21:30
  • @user1074069 No. I have it compiling to 64-bit. – J Weezy Dec 10 '18 at 21:35
  • Why this limit also exists in 64-bit mode is a bit mysterious, I've haven't yet seen a compelling explanation for it. Could be an architectural limitation, but I strongly suspect that Microsoft left this in to stop programmers from making this big mistake. You are doubling the RAM demand by creating this view. Files are already memory-mapped by design, the file system cache takes care of it. Doing it again is a bad idea, especially for such a huge view. The RAM needed to map the view will be taken away from the RAM used by the file system cache, in effect making it much less effective. – Hans Passant Dec 10 '18 at 22:16
  • @HansPassant OK, but how do I get around that? What is the most efficient way to solve this? If you know the answer please provide. Otherwise, do you know of any tutorials online I can use? – J Weezy Dec 10 '18 at 22:19
  • It is simple, throw a bunch of code away. Use FileStream or StreamReader, they know how to use the file system cache without any help. Setting system requirements isn't going to hurt, you'd like 32GB of RAM if you are using FileStream.Seek() a lot. If you only use sequential access then don't worry about it. – Hans Passant Dec 10 '18 at 22:30
  • @HansPassant That appears to be the answer. Oddly enough, the MemoryMappedViewStream error has gone away. Even stranger, I have an enormous amount of tabs in Chrome open, so my memory usage has gone from 5GB to 12GB (out of 16GB total). Anyways, if StreamReader can handle the same file size with less complexity and overhead, then I will accept that as the answer - if you want to post it. – J Weezy Dec 10 '18 at 22:54

1 Answers1

0

A memory mapped view stream is a stream over a memory mapped view. It does not provide a stream on the whole file, but just the part that you map. You will still need to map the file in chunks to read the whole thing. Unless you really need shared memory you are probably just better off with reading the file in chunks.

Mike
  • 3,462
  • 22
  • 25
  • Any idea how I would do that? I am used to reading large files through SSIS connection managers, which abstract away the reading of files. In this case, I cannot do that because the file contains records that are of different length (I am parsing the file into something SSIS connection managers can use). – J Weezy Dec 10 '18 at 21:01
  • Unless you really need to write the code yourself, I would probably just use another log file parser tool (like MSFT logparser.exe) to do this. It provides the ability to query logs with a SQL like syntax and already handles large files very well. If you still feel that you want to write the code then we would need more information about what you want to do when parsing the logs (like structure and types of data you are searching /filtering for) in order for anyone to provide you code that meets your needs. – Mike Dec 11 '18 at 15:52
  • This seems like an intermittent issue. I am able to use StreamReader as Hans has suggested. I went ahead and marked this answered as you have provided some useful information. Thanks. – J Weezy Dec 11 '18 at 21:29