Byte[stream.length] - out of memory exception, best way to solve?

Question

i am trying to read the stream of bytes from a file. However when I try to read the bytes I get a

The function evaluation was disabled because of an out of memory exception

Quite straightforward. However, what is the best way of getting around this problem? Is it too loop around the length at 1028 at a time? Or is there a better way?

The C# I am using

BinaryReader br = new BinaryReader(stream fs);

// The length is around 600000000
long Length = fs.Length;

// Error here
bytes = new byte[Length];

for (int i = 0; i < Length; i++)
{
   bytes [i] = br.ReadByte();
}

Thanks

Do you really need to have _all_ the data in memory? What is the use case? — ken2k, Jun 10 '14 at 13:18
You'll have very low odds for allocation 600 Megabytes in a 32-bit process. Hard to find a hole in the address space that is *that* big unless you do so *immediately* after starting the program. Surely there's a better solution for this, you don't give us a chance to guess at one. You'll need a 64-bit operating system or use a memory-mapped file. — Hans Passant, Jun 10 '14 at 13:19
It's quite straightforward: you've run out of memory (2GB) that can be addressed in .net. Probably, you should not try to allocate 600M in the `bytes = new byte[Length];` line. What do you want to do with these data? — Dmitry Bychenko, Jun 10 '14 at 13:19
This is a debugger message. It does not mean that your code ran out of memory. — usr, Jun 10 '14 at 13:21
The question is how do you use this array? Do you really need an array? — Magnus, Jun 10 '14 at 13:26
It's not the question but reading a file byte by byte is probably also not the best solution. — Dirk, Jun 10 '14 at 13:45

score 4 · Accepted Answer · edited Nov 30 '19 at 05:49

4

Well. First of all. Imagine a file with the size of e.g. 2GB. Your code would allocate 2GB of memory. Just read the part of the file you really need instead of the whole file at once. Secondly: Don't do something like this:

for (int i = 0; i < Length; i++)
{
   bytes [i] = br.ReadByte();
}

It is quite inefficient. To read the raw bytes of a stream you should use something like this:

using(var stream = File.OpenRead(filename))
{
    int bytesToRead = 1234;
    byte[] buffer = new byte[bytesToRead];

    int read = stream.Read(buffer, 0, buffer.Length);

    //do something with the read data ... e.g.:
    for(int i = 0; i < read; i++)
    {
        //...
    }
}

edited Nov 30 '19 at 05:49

Community

1
1

answered Jun 10 '14 at 13:20

Florian

5,918
3
47
86

A much moe elegant way, thank you. I am trying to pass back a byte[] array so I can store this in a database ( to read the item back at a later date )is it possible using the above code to do this? – user3428422 Jun 10 '14 at 13:54
Well. I am not sure whether it is a good idea to store such large data blobs in a database. But if you really want to do this, you have to split up the data. Read a part of the file, commit to the database, read the next part and commit it again. – Florian Jun 10 '14 at 13:59
Yeah I thought that may be the answer. shame you cant append bytes but thanks for the code! – user3428422 Jun 10 '14 at 14:02

score 0 · Answer 2 · answered Jun 10 '14 at 13:18

When you try to allocate an array, the CLR lays it out contiguously in virtual memory given to it by the OS. Virtual memory can be fragmented, however, so a contiguous 1 GB block may not be available, hence the OutOfMemoryException. It doesn't matter how much physical RAM your machine has, and this problem is not limited to managed code (try allocating a huge array in native C and you'll find similar results).

Instead of allocating a huge array, I recommend using several smaller arrays, an ArrayList or List, that way the Framework can allocate the data in chunks.

Hope that helps

score 0 · Answer 3 · answered Jun 10 '14 at 13:21

0

I believe that the instantiation of the stream object already reads the file (into a cache). Then your loop copies the bytes in memory to another array.

So, why not to use the data into "br" instead of making a further copy?

answered Jun 10 '14 at 13:21

Dario

11
4

The BinaryReader does not load everything into memory, it works by loading small chunks of data when it's needed. – Bun Jun 10 '14 at 13:25

Byte[stream.length] - out of memory exception, best way to solve?

3 Answers3

Linked