Dynamic (type) binary buffer in C++?

Question

Currently, I'm using std::vector<char> like this:

char data, data2, data3;

vec.push_back(data);
vec.push_back(data2);
vec.push_back(data3);

However, since I'm storing binary data, sometimes I need to push data of different sizes (i.e. not a single char), and I need to manually split that data into single bytes, which is not convenient or readable.

Something like this would be perfect:

buffer.push<int>(some_int); // some_int = 0xAABBCCDD
buffer.push<char>(some_char); // some_char = 0xFF
buffer.push("arbitrary_data", arbitrary_length);

The resulting memory would be:

AA BB CC DD FF ...........
// ^ int
//          ^ char

Is there any standard way of doing it, or I need libraries / own implementation?

While this is not the best way to go about it, you can always use `std::vector`. — Catalin, Sep 14 '15 at 06:22
You could use a vector of unsigned char, resize it whenever you need a new item, and construct that type in the array. — Weak to Enuma Elish, Sep 14 '15 at 06:33
There's no standard way for this. `boost::any` is handy for something like what you're trying to do. — eerorika, Sep 14 '15 at 06:36
Did you looked to [MessagePack](http://msgpack.org/) ? It could serialize int, string or structure, then you could add it to an `std::vector` — mpromonet, Sep 14 '15 at 06:59
@JamesRoot: Except that the location would likely be unaligned. — MSalters, Sep 14 '15 at 07:41
@MSalters how do you align a single-byte vector? without wasting 3 bytes per byte, that is — rev, Sep 14 '15 at 07:42
@AcidShout: You don't. Which is why James's comment and KemyLand's answer are wrong. BTW, don't assume that alignment is always to 4 bytes. Check `alignof(T)`. On a related note, don't assume endianness either. You may very well get `0xDD 0xCC 0xBB 0xAA` after pushing `0xAABBCCDD`. — MSalters, Sep 14 '15 at 07:45
Without knowing your use-case, it feels like you might want to use `stringstream` instead of `vector`. — , Sep 14 '15 at 07:49
@MSalters @Hurkyl I'm working with a propietary file format, which is like `[byte][byte][byte][byte][2 bytes][2 bytes][null-terminated string][byte][4 bytes]`... which is why I need something like this (i.e. alignment would break the format) — rev, Sep 14 '15 at 07:51
@AcidShout: That is a valid file format, but probably a valid format for objects in memory. Not a big deal, though, you can memcpy all the values into the `std::vector`. Don't forget `htonl`/`htons` or other endian-fixing functions! — MSalters, Sep 14 '15 at 10:09
@MSalters yeah, I see, but how do I properly align that? Or it just can't be aligned and that's it? — rev, Sep 14 '15 at 10:10
I just wrote something similar to the accepted answer that has aligned access at the cost of speed (ironically). — Weak to Enuma Elish, Sep 14 '15 at 10:11
Actually I can probably rewrite some of it to mitigate the speed. It just has to keep track of padding in between values so that it can put on the facade of a normal vector. For example `for (int i = 0; i < aligned.size(); ++i) std::cout << aligned[i];` would print like a normal vector or array, even if there was padding between every value you tried to access. — Weak to Enuma Elish, Sep 14 '15 at 10:18

score 1 · Answer 1 · answered Sep 14 '15 at 07:46

1

What you're looking for is called serialization, and it's not part of the ISO standard. Boost does have a library for it, though.

answered Sep 14 '15 at 07:46

MSalters

173,980
10
155
350

score 0 · Answer 2 · answered Sep 14 '15 at 06:37

You could use pointer arithmetic in combine with reinterpret_cast like the example bellow:

std::vector<char> v;
v.push_back('a');
v.push_back('b');
int a = 20;
v.insert(v.end(), reinterpret_cast<char*>(&a), reinterpret_cast<char*>(&a) + sizeof(a));

or if you have a class/struct:

struct foo {int a = 1 , b = 2, c = 3;};

std::vector<char> v;
foo b;
v.insert(v.end(), reinterpret_cast<char*>(&b), reinterpret_cast<char*>(&b) + sizeof(b));

3442 · Accepted Answer · 2015-09-14T08:37:19.710

What about this? It works even for non-POD types! (Note: C++11 code!)

#include <new>
#include <vector>
#include <utility>

class BinaryVector : public std::vector<unsigned char> {
    public:
        template<typename T>
        void push(const T &what) {
            this->resize(this->size() + sizeof(T));
            new(this->data() + this->size() - sizeof(T)) T(what);
        }

        template<typename T>
        void push(T &&what) {
            this->resize(this->size() + sizeof(T));
            new((T*)(this->data() + this->size() - sizeof(T))) T(XValue(what));
        }

        template<typename T>
        T pop() {
            T tmp(std::move(*(T*)(this->data() + this->size() - sizeof(T))));
            ((T*)(this->data() + this->size() - sizeof(T)))->~T();
            this->resize(this->size - sizeof(T));

            return tmp;
        }
};

Creating new objects without allocating memory yourself, and letting the vector handle it instead. Genius! — rev, Sep 14 '15 at 07:21
Be careful with memory alignment ! [Do I really have to worry about alignment when using placement new operator?](http://stackoverflow.com/a/11782277/1743220) -> Yes — Thomas B., Sep 14 '15 at 07:30

Dynamic (type) binary buffer in C++?

3 Answers3