C# increase the size of reading binary data

Question

I am using the below code from Jon Skeet's article. Of late, the binary data that needs to be processed has grown multi-fold. The binary data size file size that I am trying to import is ~ 900 mb almost 1 gb. How do I increase the memory stream size.

public static byte[] ReadFully (Stream stream)
{
    byte[] buffer = new byte[32768];
    using (MemoryStream ms = new MemoryStream())
    {
        while (true)
        {
            int read = stream.Read (buffer, 0, buffer.Length);
            if (read <= 0)
                return ms.ToArray();
            ms.Write (buffer, 0, read);
        }
    }
}

A stream doesn't take up memory. The byte array, of which you are making with `ms.ToArray()`, is taking up memory. Don't return a `byte[]` if you don't want _all_ of the file data loaded into memory. — gunr2171, Jan 30 '18 at 18:56
That solution reads the entire file into memory. It sounds like you should explore a solution where you don't have to have the entire file in memory. — juharr, Jan 30 '18 at 18:57
I believe you may be using the *ReadFully* method incorrectly. Why do you need to read the whole stream into an array to process it? The *Stream* class already offers seeking. — IS4, Jan 30 '18 at 18:58
the memorystream will automatically grow, it will run out of memory eventually tho. Look at this - a chunked MemoryStream class https://stackoverflow.com/questions/1203121/chunked-memorystream — pm100, Jan 30 '18 at 19:01

score 0 · Answer 1 · answered Jan 30 '18 at 19:03

Your method returns a byte array, which means it will return all of the data in the file. Your entire file will be loaded into memory.

If that is what you want to do, then simply use the built in File methods:

byte[] bytes = System.IO.File.ReadAllBytes(string path);
string text = System.IO.File.ReadAllText(string path);

If you don't want to load the entire file into memory, take advantage of your Stream

using (var fs = new FileStream("path", FileMode.Open))
using (var reader = new StreamReader(fs))
{
    var line = reader.ReadLine();
    // do stuff with 'line' here, or use one of the other
    // StreamReader methods.
}

score 0 · Answer 2 · 2018-01-30T19:11:51.023

You don't have to increase the size of MemoryStream - by default it expands to fit the contents.

Apparently there can be problems with memory fragmentation, but you can pre-allocate memory to avoid them:

using (MemoryStream ms = new MemoryStream(1024 * 1024 * 1024))  // initial capacity 1GB
{
}

In my opinion 1GB should be no big deal these days, but it's probably better to process the data in chunks if possible. That is what Streams are designed for.

C# increase the size of reading binary data

2 Answers2