Container to present multiple memory chunks as single continuous one

Question

Is there some "standard" container (STL, boost) which can present multiple memory chunks as single continuous one? I need to work with some data with following conditions:

Total size of data is not known in advance (web response)
Memory is allocated in chunks (with some external allocation function, which I have no control of)
Memory freeing is not controlled by me, so reallocations are relatively expensive

So, after getting all the data, I have a list of memory chunks. And I need to apply some STL algorithms (search, copy, etc.) to data as a whole. There is a solution to write container to hold info about these chunks + forward iterator which is able to "jump" from one chunk to another.

But problem seems rather general, so I hope that there is some well-known answer which I'm missing. Thanks in advance.

score 4 · Accepted Answer · answered Mar 19 '12 at 09:10

4

The memory is provided to you, you say. That sounds like you don't want to copy it. No problem, the STL philosophy is quite flexible. You don't actually need a container; they're just there for memory management and that's already taken care of.

What you do need is an iterator. There's no standard one; you'll have to write one yourself. There are just too many slight variations to provide a standard solution for this. But don't worry, it's fairly easy. You get the necessary typedefs if you inherit from std::iterator<value_type>, so you need to only write operator* (straightforward) and operator++/operator--/operator+/operator- (which understand the chunks).

answered Mar 19 '12 at 09:10

MSalters

173,980
10
155
350

I totally agree that I need an iterator and I've mentioned it as a possible solution. But using an iterator without a container looks strange for me. Because I should get my `begin()` and `end()` somehow, so encapsulating this stuff in container seems natural. Maybe word _container_ is misused here by me. I'm not talking about STL containers concept but about some "holder" for separate chunks which would encapsulate them. I just had hope that I shouldn't implement this stuff myself. Anyway, thanks for an answer. – iw.kuchin Mar 20 '12 at 04:07
"_Container_" is well defined in C++ (it's chapter 23 of the standard). – MSalters Mar 20 '12 at 08:24

score 0 · Answer 2 · edited May 23 '17 at 10:34

0

I needed something similar and after having a look around ended up writing my own, based on overloading the [] operator. For the related Q&As see the following threads;

Can I use the [] operator in C++ to create virtual arrays

Good C++ array class for dealing with large arrays of data in a fast and memory efficient way?

edited May 23 '17 at 10:34

Community

1
1

answered Mar 19 '12 at 08:34

SmacL

22,555
12
95
149

`operator[]` is entirely unrelated to algorithms such as `std::copy` and `std::search`. – MSalters Mar 19 '12 at 09:11

score 0 · Answer 3 · answered Mar 19 '12 at 09:01

So, after getting all the data, I have a list of memory chunks. And I need to apply some STL algorithms (search, copy, etc.) to data as a whole. There is a solution to write container to hold info about these chunks + forward iterator which is able to "jump" from one chunk to another.

It sounds like you need an iterator that traverses all your chunks seamlessly. std::deque<> provides a similar iterator because it also allocates memory in chunks.

Unless you really need it to be in one contiguous block of memory. In this case all chunks would need to copied into one continuous piece of memory.

Container to present multiple memory chunks as single continuous one

3 Answers3