MemoryStream exhausting

Question

I am writing some bytes into a MemoryStream. I have given it a size of 100 MB

  using (MemoryStream Stream = new MemoryStream())
        {             
                for ( loop 1400 time)
                {                        
                  func(i, OStream, Stream, );

                }

                lock (this)
                {                       
                 Stream.WriteTo(OStream);
                }

}

func(i, OStream, Stream)
{
  loop 19 times
      get buffer;
   Stream.Write(buffer, 0, buffer.Length);
}

Buffer is a byte[] with size varying from 65536 to 116454. I loop 1000 times. I am writing it in for loop, but i am getting OutOfMemoryException exception.

Is there any way expanding MemoryStream and avoid the exception?

*"I loop 6000 times"* - where? Can you show the loop perhaps the problem is there. `100mb` doesn't looks to me as too much. — Sinatr, Mar 23 '17 at 10:43
Why are you setting the capacity of your memory stream to 100MB when you already know that you're going to write much more to it? Your setup guarantees that the stream will have to expand its storage at least once (if not multiple times), which involves copying its content to a new array. That (briefly) requires more than twice as much memory as the stream's current capacity. — Pieter Witvoet, Mar 23 '17 at 11:00
@PieterWitvoet i tried allocating 1 GB but got exception of outof memory at onset of app. — murmansk, Mar 23 '17 at 11:06
With the update given, you loop 3 times 6000 times, which is 18000 times. Also, this looks far from C#. — Thomas Weller, Mar 23 '17 at 11:09
But then, the real question is: why do you need a `MemoryStream`? Perhaps you could do "it" (whatever "it" is) in another way. — xanatos, Mar 23 '17 at 11:10

score 5 · Accepted Answer · edited May 23 '17 at 11:46

Considering the amout of data, that's minimum 6000*65536 bytes, which is <400 MB for the byte arrays and maximum 6000*116454 bytes, which is <700 MB. The MemoryStream will double its size whenever necessary, so it first has 100 MB, then 200 MB, then 400 MB, perhaps 800 MB.

Even if the garbage collector cannot clean up in between, you might think that this is a maximum total of 2200 MB (700 for byte arrays, 100 for the initial memory stream, 200 for the first expansion, ...). In fact, that may be the case and "enough" memory is left. However, it may not be available in a contiguous block.

What you're likely facing is a memory fragmentation issue. A large number of your byte arrays tends to exceed 85001 bytes, so they are allocated on the large object heap. The large object heap does not get compacted and thus suffers from fragmentation.

You can try to use the answer at How to use WinDbg to track down .NET OutOfMemoryExceptions. I also posted a memory fragmentation example in case you want to compare your situation to it.

Potential fixes

A quick fix would be to compile as x64, because you have way more memory available there. A cleaner approach depends on your real situation and requires an optimization of the algorithm, e.g. you could calculate the total size of the needed result in advance, so that the MemoryStream does not need to double its size.

It might also be good to use a single 200 kB byte array and re-fill it with temporary data instead of creating new arrays all over the place.

If your final task is to write the data into a file, use a FileStream and write it directly into a file.

You might also get rid of the MemoryStream completely and just keep all byte arrays together in a List<byte[]>.

And probably many other solutions...

There is another solution that you can add: `List` – xanatos Mar 23 '17 at 11:16 — xanatos, Mar 23 '17 at 11:16

MemoryStream exhausting

1 Answers1

Potential fixes