An effective way to concatenate multiple buffers

Question

I need to design an efficient and readable class with 2 main functions:

add_buffer(char* buffer) - add a buffer.
char* read_all() - get one big buffer that contains all the buffers that the user added until now (by order).

for example:

char first_buffer[] = {1,2,3};
char second_buffer[] = {4,5,6};

MyClass instance;
instance.add_buffer(first_buffer);
instance.add_buffer(second_buffer);
char* big_buffer = instance.read_all(); // big_buffer = [1,2,3,4,5,6]

NOTE: There are a lot of solutions for this problem but I'm looking for an efficient one because in real life the buffers will be many and big, and I want to save a lot of copying and reallocs (like what std::vector does). I'm also want a readble c++ code.

NOTE: The real life problem is: I'm reading data from an HTTP request that came to me at separated chunks. After all chunks arrived I want to return the whole data to the user.

Unless you've preallocated a ton of memory, you probably can't avoid several copying reallocs. — Cornstalks, Dec 29 '15 at 15:33
is it neasccery that the output buffer to have contiguous memory? — Humam Helfawi, Dec 29 '15 at 15:34
I'm reading data from an HTTP request that came to me at separated chunks. After all chunks arrived I want to return the whole data to the user. — Ofer, Dec 29 '15 at 15:39
***So, what is your question?*** Questions usually end with Question Marks (**?**) — abelenky, Dec 29 '15 at 15:49
[`boost::algorithm::join`](http://stackoverflow.com/a/1833499) — GingerPlusPlus, Dec 29 '15 at 19:46

YSC · Accepted Answer · 2015-12-29T15:36:41.383

8

Use an std::vector<char> with enough memory reserved. Since C++11, you can access the internal buffer with std::vector::data() (until C++11, you have to use &*std::vector::begin()).

edited Dec 29 '15 at 15:36

answered Dec 29 '15 at 15:33

YSC

38,212
9
96
149

score 2 · Answer 2 · edited May 23 '17 at 12:31

If you can use Boost, boost::algorithm::join will do:

#include <boost/algorithm/string/join.hpp>
#include <vector>
#include <iostream>

int main(int, char **)
{
    std::vector<std::string> list;
    list.push_back("Hello");
    list.push_back("World!");

    std::string joined = boost::algorithm::join(list, ", ");
    std::cout << joined << std::endl;
}

Output:

Hello, World!

Original answer by Tristram Gräbener

score 0 · Answer 3 · edited Dec 29 '15 at 15:48

0

Use some standard approach like,

start with some initial memory, say 256
whenever it gets full use reallocate and double the size.

If you don't want to do it yourself, use STL containers like

std::vector<char>

It automatically reallocates memory for you when buffer is full.

edited Dec 29 '15 at 15:48

Lightness Races in Orbit

378,754
76
643
1,055

answered Dec 29 '15 at 15:37

Abhishek Kumar

169
7

2

Dangerous startegy if implemented exactly as written. – SergeyA Dec 29 '15 at 15:40
3

This is a pretty poor suggestion which will result in unacceptable and needless reallocations. – Lightness Races in Orbit Dec 29 '15 at 15:48
Even std::vector uses the same strategy.If anybody has any other strategy, please comment. – Abhishek Kumar Dec 29 '15 at 15:51

An effective way to concatenate multiple buffers

3 Answers3