System.OutOfMemory exception when trying to read large files

Question

public static byte[] ReadMemoryMappedFile(string fileName)
    {
        long length = new FileInfo(fileName).Length;
        using (var stream = File.Open(fileName, FileMode.OpenOrCreate, FileAccess.Read, FileShare.ReadWrite))
        {
            using (var mmf = MemoryMappedFile.CreateFromFile(stream, null, length, MemoryMappedFileAccess.Read, null, HandleInheritability.Inheritable, false))
            {
                using (var viewStream = mmf.CreateViewStream(0, length, MemoryMappedFileAccess.Read))
                {
                    using (BinaryReader binReader = new BinaryReader(viewStream))
                    {
                        var result = binReader.ReadBytes((int)length);
                        return result;
                    }
                }
            }
        }
    }

OpenFileDialog openfile = new OpenFileDialog();
        openfile.Filter = "All Files (*.*)|*.*";
        openfile.ShowDialog();
        byte[] buff = ReadMemoryMappedFile(openfile.FileName);
        texteditor.Text = BitConverter.ToString(buff).Replace("-"," "); <----A first chance exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll

I get a System.OutOfMemory exception when trying to read large files. I've read a lot for 4 weeks in all the web... and tried a lot!!! But still, I can't seem to find a good solution to my problem. Please help me..

Update

public byte[] FileToByteArray(string fileName)
    {
        byte[] buff = null;
        FileStream fs = new FileStream(fileName,
                                       FileMode.Open,
                                       FileAccess.Read);
        BinaryReader br = new BinaryReader(fs);
        long numBytes = new FileInfo(fileName).Length;
        buff = br.ReadBytes((int)numBytes);
        //return buff;
        return File.ReadAllBytes(fileName);
    }

OR

public static byte[] FileToByteArray(FileStream stream, int initialLength)
    {
        // If we've been passed an unhelpful initial length, just
        // use 32K.
        if (initialLength < 1)
        {
            initialLength = 32768;
        }
        BinaryReader br = new BinaryReader(stream);
        byte[] buffer = new byte[initialLength];
        int read = 0;

        int chunk;
        while ((chunk = br.Read(buffer, read, buffer.Length - read)) > 0)
        {
            read += chunk;

            // If we've reached the end of our buffer, check to see if there's
            // any more information
            if (read == buffer.Length)
            {
                int nextByte = br.ReadByte();

                // End of stream? If so, we're done
                if (nextByte == -1)
                {
                    return buffer;
                }

                // Nope. Resize the buffer, put in the byte we've just
                // read, and continue
                byte[] newBuffer = new byte[buffer.Length * 2];
                Array.Copy(buffer, newBuffer, buffer.Length);
                newBuffer[read] = (byte)nextByte;
                buffer = newBuffer;
                read++;
            }
        }
        // Buffer is now too big. Shrink it.
        byte[] ret = new byte[read];
        Array.Copy(buffer, ret, read);
        return ret;
    }

I still get a System.OutOfMemory exception when trying to read large files.

Given that you're reading the whole thing into memory, it's not clear why you're using a memory-mapped file... and you're not only reading it into memory, but you're converting it into hex, and then trying to display it. That will take a lot of memory. How big is the file? — Jon Skeet, Jun 28 '14 at 07:56
I have big file > 4GB and I want to read it.How to read large file into byte array to hex ? — FetFat, Jun 28 '14 at 07:57
You should work with streams rather than byte[]. So make the function return a stream instead. — Magnus, Jun 28 '14 at 08:14
_How to read large file into byte array to hex?_ Not. Hex is aimed at humans, nobody can read an 8 billion string in their life time. — H H, Jun 28 '14 at 08:21
@FetFat The question is what do you do with the byte[] after calling this function? — Magnus, Jun 28 '14 at 08:24
actually I want to create a hex viewer which can read a large file into a hex value — FetFat, Jun 28 '14 at 08:28

score 1 · Answer 1 · answered Jun 28 '14 at 08:32

1

You need to read the file in chunks, keep track of where you are in the file, page the contents on screen and use seek and position to move up and down in the file stream.

answered Jun 28 '14 at 08:32

mtmk

6,176
27
32

score 1 · Accepted Answer · answered Jun 28 '14 at 08:55

If your file is 4GB, then BitConverter will turn each byte into XX- string, each char in string is 2 bytes * 3 chars per byte * 4 294 967 295 bytes = 25 769 803 770. You need +25Gb of free memory to fit entire string, plus you already have your file in memory as byte array.

Besides, no single object in a .Net program may be over 2GB. Theoretical limit for a string length would be 1,073,741,823 chars, but you also need to have a 64-bit process.

So solution in your case - open FileStream. Read first 16384 bytes (or how much can fit on your screen), convert to hex and display, and remember file offset. When user wants to navigate to next or previous page - seek to that position in file on disk, read and display again, etc.

score 0 · Answer 3 · answered Jun 28 '14 at 12:23

0

You will not be able to display 4Gb file reading all of it in memory first by any approach.

The approach is to virtualize the data, reading only the visible lines when user scrolls. If you need to do a read-only text viewer then you can use WPF ItemsControl with virtulizing stack panel and bind to custom IList collection which will lazily fetch lines from the file calculating file offset by for the line index.

answered Jun 28 '14 at 12:23

Alex des Pelagos

1,170
7
8

im using wpf ,can you tell me how to implement virtulizing stack panel and bind to custom IList collection? I'm new to WPF ,thanks – FetFat Jun 28 '14 at 19:19
Here is how to use virtuzing stack panel in ItemsControl http://stackoverflow.com/questions/2783845/virtualizing-an-itemscontrol - that will virtualize text lines. To virtualize the data you need to derive a class from IList and in indexer this[int index] get - get navigate by offset to the position in the file you need to display, read the data into the buffer exactly as line width and return the hex string of it. – Alex des Pelagos Jun 29 '14 at 20:36

System.OutOfMemory exception when trying to read large files

3 Answers3