Difference between buffer size when streams copying

Question

I saw a lot of examples of CopyStream implementation but I have question about buffer size when we copy streams.
Sample of one of CopyStreams implementation:

private void ReadWriteStream(Stream readStream, Stream writeStream)
{
    int Length = 256;
    Byte[] buffer = new Byte[Length];
    int bytesRead = readStream.Read(buffer, 0, Length);
    // write the required bytes
    while (bytesRead > 0)
    {
        writeStream.Write(buffer, 0, bytesRead);
        bytesRead = readStream.Read(buffer, 0, Length);
    }
    readStream.Close();
    writeStream.Close();
}

Questions are:

What buffers Length should be (I've seen 256, 8 * 1024, 32768)?
How different buffer size affects performance, memory usage etc.?

Related questions:
File IO with Streams - Best Memory Buffer Size - nice File IO answer. But what about in memory copying?

My case:
There is MemotyStream which I create using ClosedXML workbook.SaveAs(memoryStream); and it allocates huge amount of memory in managed heap. I've looked into sources and found that there is CopyStream method that uses 8 * 1024 buffer size. Could changing this size somehow decrease memory usage?
Note: Stream takes almost 1Gb of memory.

In your case, might it be that majority of memory is used by memory stream rather than copy procedure? Worth checking because 8KB buffer is tiny for streams of hundreds of MBs. — Maciej, Feb 18 '12 at 12:37
What's the size of your XML document (when written to disk)? If it's 1GB, then you should expect the memory stream to be 1GB... or where do you expect the data to go if not - well - to the (managed) memory? — thoean, Feb 18 '12 at 13:21
@Markus Excel file itself takes only 20MB. And it's definitely a bad sign. — Anatolii Gabuza, Feb 18 '12 at 13:33
"Stream takes almost 1GB of memory?" Does that mean, that the stream's size is that large or only that your application uses 1GB additional memory after saving? Did you debug ClosedXML internally? Set a breakpoint before the (internal) save method and figure out if the memory is increased exactly there (or find out the offending call). Alternatively you might want to attach a profiler. — thoean, Feb 18 '12 at 18:17

Maciej · Accepted Answer · 2012-02-18T12:50:01.397

6

If you are using .NET 4 you can do it simpler:

srcStream.CopyTo(dstStream);

But if you want/need to implement it yourself I would suggest smaller buffer (256B - 1KB) for memory streams and medium size buffer (10KB) for file streams. You can also make it dependent on size of the source stream for example 10% with some size limit of 1MB or so.

For files, the bigger the buffer the faster copy operation (to a degree) but less safe. For memory streams small buffer is pretty much just as effective as big one but easier on memory (if you copy a lot).

edited Feb 18 '12 at 12:50

answered Feb 18 '12 at 11:58

Maciej

7,871
1
31
36

1

You're right. But I need to know how buffer size affects on memory usage and performance. – Anatolii Gabuza Feb 18 '12 at 12:06
Make some tests then. If you copy a lot and want to keep memory usage low go for small buffer first, test, increase buffer and test again to see the impact. Keep doing that until your sweet-spot. – Maciej Feb 18 '12 at 12:33

Difference between buffer size when streams copying

1 Answers1