0

Suppose I have a byte[] (buffer) and I need to keep it in memory. It also needs growing strategy x2. So each time I need it to grow I have to create another byte[] twice large and copy elements there.

During this operation the app memory usage is original_size * 3 because I have in memory both the original buffer and the new buffer.

When working with buffers of size near 300 mb this operation takes 900 mb of memory and can easily lead to OutOfMemoryException in 32-bit app.

The only way to make this better I know is to choose another growing strategy: after some constant size make the buffer grow linearly (e.g. +5mb each time). But it still requires 600 mb for my 300 mb of data!

So can I do anything about it?

I think about structure which maintains a list of buffers inside so it doesn't need to copy anything when growing - just adds new buffer to the list. It should provide methods to perform operations with those buffers as with one big buffer. Is there something in .NET like this? Or usually what's it called like?

Added:

  1. I don't know the capacity preliminary.
  2. The buffers are used in serialization process.
  3. Sorry for misleading, the question is about buffer growing problem in generic - not about solving a particular app issue.
Vlad
  • 3,001
  • 1
  • 22
  • 52
  • Have you tried with `list`? – erikscandola Jan 06 '16 at 08:19
  • You can use a `List` with the overloaded constructor which specifies the capacity. – Yuval Itzchakov Jan 06 '16 at 08:19
  • 3
    Well what are these buffers used for? What's responsible for the growing strategy? Can you judge how much data you'll need beforehand? Could you avoid having all the data in memory at once in the first place? What's consuming this data, and does it really need a single array? We really don't have enough context to help. – Jon Skeet Jan 06 '16 at 08:22
  • (And no, I don't believe .NET provides anything exactly like the buffer list you're describing, but it sounds like it *may* be a plausible solution for you... but again, we don't have enough context to say for sure.) – Jon Skeet Jan 06 '16 at 08:24
  • @JonSkeet ok, updated the question – Vlad Jan 06 '16 at 08:27
  • "used in serialization process" doesn't really give us much information. That *sounds* like it's something that could be streamed instead... I would expect that if you were changing the code to use a new data structure, you could make the change to use a streaming solution about as easily. – Jon Skeet Jan 06 '16 at 08:33
  • One more thought: if your app blows up with 900MB of data and you've currently got 300MB, how convinced are you that you'll never need to *actually* handle 900MB? – Jon Skeet Jan 06 '16 at 08:37
  • @JonSkeet you are right, in practical sense this part should be done with streams and I'll definitely check how I can alter existing code for it. But after facing this problem yesterday I'm interested to find a generic way of how can it be possible at all to directly achieve more effective buffer management. May be there is some unmanaged way? Or I don't know some suitable structures? – Vlad Jan 06 '16 at 08:56
  • Unfortunately, it's hard to give a *specific* solution to a *generic* (underspecified) problem. Typically for a sequence of bytes, you'd want some sort of stream-like API, whereas for individual objects you'd want something more like `IList`. I can imagine a `List`-like class which didn't consist of a single array but a linked list of arrays, and I can imagine a `MemoryStream`-like class implemented similarly... but they'd have quite different APIs. – Jon Skeet Jan 06 '16 at 09:09
  • @JonSkeet ok, then I'm going to edit the question to mention only byte buffers, not arrays. – Vlad Jan 06 '16 at 09:15

1 Answers1

0

In this scenario I think you should use ArrayList or List instead of array. List is better than ArrayList. For more information visit:

Which is better? array, ArrayList or List (in terms of performance and speed)

Community
  • 1
  • 1
Mahedee
  • 166
  • 7
  • You do realize that List and ArrayList internally have the same growing code for its internal array, right? – Vlad Jan 06 '16 at 10:56