SystemOutOfMemory exception when calling "File.ReadAllBytes();"

Question

I'm calling File.ReadAllBytes(PATH);, the file I'm loading is very large (about 2GB), I'm getting a SystemOutOfMemory Exception, how can I solve this?? NOTE: the program is made to load large files! Thanks in advance.

EDIT: I'm trying to get those bytes, then compress them in some way, then convert them to a string, then reconvert the string to a byte array then to a file.

Show your code. If you have to load whole file and don't have available memory available, exception will be thrown. — kamil-mrzyglod, Feb 21 '16 at 17:06
What architecture are you compiling it for? Look at this: http://stackoverflow.com/questions/200348/is-there-a-memory-limit-for-a-single-net-process — Paweł Mach, Feb 21 '16 at 17:07
Do you actually *need* to load the entire file into memory? Can you stream it instead? What are you doing? — Charles Mager, Feb 21 '16 at 17:07
Hello and welcome. Please show the code you are using. For very large files as you describe, I'd recommend that you probably not use the `ReadAllBytes()` method and use a `StreamReader` instead. — Lemonseed, Feb 21 '16 at 17:08
Try to compile your app as 64 bit (uncheck prefer 32 bit on build tab in project properties) + set amd64 as platform. — csharpfolk, Feb 21 '16 at 17:08
@Dave: `StreamReader` would be for text; the OP is trying to read binary data. — Jon Skeet, Feb 21 '16 at 17:09
@Dave Plain `Stream` (`FileStream` specifically) directly without any helper reader classes should be fine. — , Feb 21 '16 at 17:16
I don't want to load some text in a file, I want to load any file (e.g. exe file) — Mohamed Mokhtar, Feb 21 '16 at 17:18
Don't do that. Read in a little bit of the file, compress it, write out a little bit of the file, repeat until you're done. Don't use strings as binary storage. — Eric Lippert, Feb 21 '16 at 17:21
Incremental approach is best in my opinion. See `BinaryReader.ReadBytes` on MSDN [here](https://msdn.microsoft.com/en-us/library/system.io.binaryreader.readbytes(v=vs.110).aspx). You can read a chunk at a time to a buffer, update progress as you go, etc. — Lemonseed, Feb 21 '16 at 17:24

score 1 · Accepted Answer · answered Feb 21 '16 at 17:29

Approach using BinaryReader as seen on MSDN:

const int CHUNK_SIZE = 1024;

using (FileStream stream = new FileStream(filename, FileMode.Open, FileAccess.Read))
{
    using (BinaryReader reader = new BinaryReader(stream, new ASCIIEncoding()))
    {
        byte[] chunk;

        chunk = reader.ReadBytes(CHUNK_SIZE);
        while (chunk.Length > 0)
        {
            DumpBytes(chunk, chunk.Length);
            chunk = reader.ReadBytes(CHUNK_SIZE);
        }
    }
}

Where the DumpBytes() method does something useful with the bytes you have read:

public static void DumpBytes(byte[] bytes, int length)
{
    ...
}

Using this incremental approach will be more memory efficient than loading the file all at once, as well as allow you to do other things each time you read a 'chunk' of data, such as updating the user interface, etc.

SystemOutOfMemory exception when calling "File.ReadAllBytes();"

1 Answers1