2

Is there any known header-only STL-like container/allocator for appending chunks of memory to another continuos memory area until it's filled? At the moment I am using a std::vector<char> vec, because it has some useful interface, but it's not optimal and somehow I think I am abusing it for my needs.

I first use std::vector::reserve to fix its capacity and allocate the required memory once for all to avoid unnecessary reallocations and then use std::copy(&chunk[0], &chunk[size], vec.data() + vec.size()) each time to append the new chunks of memory to the unfilled memory area behind the vector (of course size() <= capacity()). After each copy I explicitly update the size of the vector accordingly. Ok, I could use a back_inserter. But this not the point now (see below).

Of course std::copy could be specialized for char by any implementation so that it can just call memcpy at the end, but this is not a guarantee. Calling memcpy by myself to append the chunk to the memory already allocated by the vector to have such guarantee is just ugly. Are there better/more elegant options?

EDIT: I have no control on how the chunks of memory are allocated. They are given.

Martin
  • 9,089
  • 11
  • 52
  • 87
  • 10
    I'd say stick with `vector` until you've proven beyond doubt that this is inferior to a hand-rolled `memcpy` solution, and *then* work on encapsulating that one into a C++ construct. – Kerrek SB Jan 25 '13 at 10:21
  • did you see this: http://stackoverflow.com/questions/3482941/how-do-you-realloc-in-c – 0x90 Jan 25 '13 at 10:28
  • And why not use a char array? – Dariusz Jan 25 '13 at 10:39
  • @0x90: that does not answer my question. There is no guarantee std::vector calls realloc on resize(). But in my case I already know the final size, so I can call reserve() instead. My concern is more about std::copy() which is not guaranteed to call memcpy. Doing memcpy by myself on the memory reserved by the vector is the only efficient solution I have found. – Martin Jan 25 '13 at 10:40
  • This question doesn't seem to be about appending chunks of memory to a continuous memory area. I think it's about copying memory. – Peter Wood Jan 25 '13 at 12:27
  • Possible duplicate: [`memcpy()` vs `std::copy()`](http://stackoverflow.com/q/4707012/1084416) – Peter Wood Jan 25 '13 at 12:33
  • Basically I'm not sure if you correctly use the concepts of containers. Containers should explicitly abstract from the underlaying implementation. What you do is assume that vector has a contiguous memory arare which is not guaranteed. – grundprinzip Jan 25 '13 at 14:31
  • @grundprinzip: please read the standard. std::vector guarantees contiguos memory. I am allowed to use the vector that way, although it's an "ugly" way, since at some point I need to access the data without using the vector interface (to make sure data is copied efficiently). – Martin Jan 25 '13 at 14:44

3 Answers3

0

Basically the only way to "really" append memory to a contiguous memory area is to use the Posix function realloc(). Even with these allocated regions you can use std::copy, you only have to maintain knowledge about the end position in your region.

However, realloc() does not guarantee that the memory area will be at the previous memory location, but rather that the new block is fully contiguous. See man page for more details.

Few things to remember:

  • If you allocated memory via posix_memalign() it is not guaranteed that the memory alignment will be kept
  • There is no way to find out if the realloc() copied the data or not.
grundprinzip
  • 2,471
  • 1
  • 20
  • 34
  • hmm, I don't think your answer answers my question. I have made it more clear. I'd really like to have a vector interface for the memory. realloc() is not point really. I could just call reserve() once for all, std::copy might be not efficient and memcpy is ugly. I have no control on how/where the chunks are allocated. – Martin Jan 25 '13 at 10:49
  • `std::copy` is quite efficient. – Puppy Jan 25 '13 at 11:38
  • @DeadMG : there is no guarantee it uses memcpy/memmove, so there is no guarantee it is efficient. – Martin Jan 25 '13 at 11:49
  • @Martin, I still don't get why you want to use a stl container for something it's not meant to. Plus with your requirements you're basically relying on hidden properties of the implementation. The allocator used by the vector could do magic that you do not know *and* the vector does not promise contiguous memory. – grundprinzip Jan 25 '13 at 14:33
  • @grundprinzip I do rely on standard guarantees. vector does provide contiguous memory. – Martin Jan 25 '13 at 14:41
0

I know it's not exactly what you want, but std::deque usually stitches together contiguous blocks of memory.

There aren't any re-allocations as it grows, but the whole data may not be contiguous.

edit

Using a vector with preallocated memory, you can insert values at the end, which won't result in a reallocation unless the new values would make the size exceed the capacity:

vector<char> values(PRE_ALLOCATED_SIZE);
// ...
values.insert(values.end(), chunk, chunk+CHUNK_SIZE);
Peter Wood
  • 23,859
  • 5
  • 60
  • 99
  • 1
    You cannot add previously allocated chunks on to an existing chunk. – Puppy Jan 25 '13 at 11:39
  • @DeadMG: what are you talking about? – Martin Jan 25 '13 at 11:50
  • @Martin, you said you wanted to append chunks of memory to another contiguous memory area. It is not possible to put two chunks of contiguous memory together so that they are contiguous (unless they happened to be contiguous to start with). – Peter Wood Jan 25 '13 at 11:54
  • @Peter : Ok. I see your point. I think I was not clear enough, I'll edit my question again. The contiguous area is fixed and preallocated once. Chunks need to be appended until this single area is filled, so that the size of the vector will grow, but not its capacity. – Martin Jan 25 '13 at 12:02
  • @Peter, I made my question clear enough. Please read it again if you can. – Martin Jan 25 '13 at 12:17
  • @PeterWood the point in using std::copy instead of std::vector::insert was that std::copy has more chances to be specialized for char than std::vector::insert. In my implementation std::copy uses memmove for std::copy, while vector::insert uses a loop (g++ -O3) – Martin Jan 25 '13 at 13:14
  • @Martin This seems to be the nub of it. Make that your question: My platform's implementation of `vector::insert` doesn't use `memcpy` or `memmove`. What's the cleanest way of inserting efficiently? – Peter Wood Jan 25 '13 at 13:21
  • @PeterWood probably I should have asked if there is a sort of STL-like buffer container with a clean interface for managing data efficiently: this is an idea, but nothing concrete actually: http://www.codesynthesis.com/~boris/blog/2011/08/09/do-we-need-std-buffer/ – Martin Jan 25 '13 at 13:31
0

This probably is not what you're looking for, since they are designed for homogenous containers (i.e. where all the elements have the same type). I don't know if that fits your use case. On the off chance that you find it useful, I point you at:

get_temporary_buffer -- allocate raw storage (there's also a way to return the storage).

raw_storage_iterator -- output iterator over raw storage.

uninitialized_copy_n -- Copy construct several objects into raw storage (there is also a single element version).

rici
  • 234,347
  • 28
  • 237
  • 341