Creating custom boost serialization output archive

Question

I'm trying to use Boost serialization to serialize objects into a buffer. The objects are large (hundreds of MB) so I don't want to use binary_oarchive to serialize them into an std::stringstream to then copy-them into their final destination. I have a Buffer class which I would like to use instead.

The problem is, binary_oarchive takes an std::ostream as parameter, and this class's stream operator is not virtual, so I can't make my Buffer class inherit from it to be used by binary_oarchive. Similarly, I haven't found a way to inherit from binary_oarchive_impl that would let me use something else than std::ostream to serialize into.

So I looked into how to create an archive from scratch, here: https://www.boost.org/doc/libs/1_79_0/libs/serialization/doc/archive_reference.html, which I'm putting back here for reference:

#include <cstddef> // std::size_t
#include <boost/archive/detail/common_oarchive.hpp>

/////////////////////////////////////////////////////////////////////////
// class complete_oarchive
class complete_oarchive : 
    public boost::archive::detail::common_oarchive<complete_oarchive>
{
    // permit serialization system privileged access to permit
    // implementation of inline templates for maximum speed.
    friend class boost::archive::save_access;

    // member template for saving primitive types.
    // Specialize for any types/templates that require special treatment
    template<class T>
    void save(T & t);

public:
    //////////////////////////////////////////////////////////
    // public interface used by programs that use the
    // serialization library

    // archives are expected to support this function
    void save_binary(void *address, std::size_t count);
};

But unless I misunderstood something, it looks like by defining my archive this way, I need to overload the save method for every single type I want to store, in particular STL types, which kind of defies the point of using Boost serialization altogether. If I don't define save at all, I'm getting compilation errors indicating that a save function could not be found, in particular for things like std::string and for boost-specific types like boost::archive::version_type.

So my question is: how would you make it possible, with Boost serialization, to serialize in binary format into a custom Buffer object, while retaining all the power of Boost (i.e. not having to redefine how every single STL container and boost type is serialized)?

This is something I've done pretty easily in the past with the Cereal library, unfortunately I'm stuck with Boost for this particular code base.

Answer 1

Don't create your own archive class. Like the commenter said, use the streambuf interface to your advantage. The upside is that things will work for any of the archive implementations, including binary archives, and perhaps more interestingly things like the EOS Portable Archive implementation.

The streambuf interface can be quite flexible. E.g. i've used it to implement hashing/equality operations:

• Generate operator== using Boost Serialization?

In that answer I used Boost Iostreams with its Device concept to make implementing things simpler.

Now if your Buffer type (which you might have shown) has an interface that resembles most buffers (i.e. one or more (void*,size) pairs), you could use existing adapters present in Boost IOstreams. E.g.

• Boost: Re-using/clearing text_iarchive for de-serializing data from Asio:receive()

where I show how to use Serialization with a re-usable fixed buffer. Here's the Proof Of Concept:

#include <boost/archive/text_iarchive.hpp>
#include <boost/archive/text_oarchive.hpp>
#include <boost/serialization/serialization.hpp>

#include <boost/iostreams/device/array.hpp>
#include <boost/iostreams/stream.hpp>
#include <sstream>

namespace bar = boost::archive;
namespace bio = boost::iostreams;

struct Packet {
    int i;
    template <typename Ar> void serialize(Ar& ar, unsigned) { ar & i; }
};

namespace Reader {
    template <typename T>
    Packet deserialize(T const* data, size_t size) {
        static_assert(boost::is_pod<T>::value     , "T must be POD");
        static_assert(boost::is_integral<T>::value, "T must be integral");
        static_assert(sizeof(T) == sizeof(char)   , "T must be byte-sized");

        bio::stream<bio::array_source> stream(bio::array_source(data, size));
        bar::text_iarchive ia(stream);
        Packet result;
        ia >> result;

        return result;
    }

    template <typename T, size_t N>
    Packet deserialize(T (&arr)[N]) {
        return deserialize(arr, N);
    }

    template <typename T>
    Packet deserialize(std::vector<T> const& v) {
        return deserialize(v.data(), v.size());
    }

    template <typename T, size_t N>
    Packet deserialize(boost::array<T, N> const& a) {
        return deserialize(a.data(), a.size());
    }
}

template <typename MutableBuffer>
void serialize(Packet const& data, MutableBuffer& buf)
{
    bio::stream<bio::array_sink> s(buf.data(), buf.size());
    bar::text_oarchive ar(s);

    ar << data;
}

int main() {
    boost::array<char, 1024> arr;

    for (int i = 0; i < 100; ++i) {
        serialize(Packet { i }, arr);

        Packet roundtrip = Reader::deserialize(arr);
        assert(roundtrip.i == i);
    }
    std::cout << "Done\n";
}