Sending an array of ints with boost::asio

Question

I want to send raw ints with boost.asio compatibly with any CPU architecture. Usually, I would convert ints to strings, but I may be able to get better performance by skipping the int/ascii conversion. I don't know what boost.asio already does under the hood such as using htonl. The documentation doesn't say and there is no way to test this code on my own PC.

Here are several methods I have to send an array of ints over the network:

1. Store the ints in an array of int32_t. Use htonl and ntohl to get correct endian-ness. If int32_t is not supported on the target machine, handle that with int_fast32_t and extra work.
2. Use boost.serialization. The only example I can find is from pre-C++11 times (see section on serialization), so I don't know if boost.serialization and boost.asio should still be used in this way. Also, as boost.serialization is a black box, it isn't clear what the performance overhead of using this library is.
3. Convert each int to the ascii representation.

I think (1) is the "correct" alternative to non-ascii conversion. Can boost.asio or boost.serialization perform any of the steps of (1) for me? If not, what is the current recommended way to send ints with boost.asio?

Answer 1

A lot will depend on what your long term needs are, i.e. what do you mean by "compatibly".

If the sending and receiving ends are guaranteed to be written in C++ then you can follow the "code first" approach. This is your options 1, 2) or 3) because (in this circumstance) you have decided that you can re-use the code. Though I would encourage using Boost serialisation, option 2), as ultimately that does a lot of the work for you (even if it might not feel like it for a simple case!).

However, if you think that, one day, a sending or receiving end might have to be written in a different language, then I would strongly encourage a "schema first" approach. This means using something like Google Protocol Buffers, or ASN.1, or XSD (XML schema + a good code generator), Thrift, Cap'n Proto, Avro (though I think Avro can be a bit awkward in non-dynamic languages like C++). There are more. With this approach you start off with a schema to define your messages (an array of ints in your case), and compile that to a language of your choice. That way, sending and receiving ends can be written in different programming languages (the schema is compiled accordingly), and will be able to talk without you having to be aware of exactly how that's managed.

Sending Over a Network

I would also encourage use of ZeroMQ, as that is a very easy way to use a network connection in a program. Being message orientated, instead of stream orientated, makes life an awful lot easier. For example, a ZeroMQ "message" can be the byte array the serialisation tech has output in serialising an object.

Plus, ZeroMQ is available in almost any language. Google Protocol Buffers - a good schema-first serialisation technology - is also available in many languages. My personal preference - ASN.1 - is a more strict serialisation (it does "constraints"), but good tools for lots of languages cost money.

The combination of schema-first serialisation, + ZeroMQ, makes it pretty easy to convey messages problem-free between almost any combination of language, OS, and platform.

Some of these serialisation techs (e.g. Google Protocol Buffer, Thrift) provide an RPC-like framework too, which can also satisfy the "sending over a network" problem.

Answer 2

The other answer by @bazza has good professional advice. I won't repeat that. I get the impression you're more focusing on understanding the implementation specifics here, so I'll dive into the details of that for you here:

---

Yeah, option 1 seems okay for the simple use case you describe.

I don't know what boost.asio already does under the hood such as using htonl

It doesn't except perhaps in the protocol layers - but those are implementation details.

The documentation doesn't say

That's because it doesn't have an opinion on your application layer. The fact that it isn't mentioned is a good sign that no magic happens.

---

Can boost.asio or boost.serialization perform any of the steps of (1) for me?

Actually, no. The Boost binary archive is not portable (see also EOS Portable Archive).

Ironically, it can portably use one of the text archive formats https://www.boost.org/doc/libs/1_46_1/libs/serialization/doc/archives.html#archive_models (XML and plain text are provided), but as you surmised they can have considerable overhead: Boost C++ Serialization overhead. Besides, there will be copying into and out of the stream format.

If not, what is the current recommended way to send ints with boost.asio?

I'd consider using the raw array/vector:

std::array<int32_t, 45> data;

boost::asio::async_write(socket_,
    boost::asio::buffer(data),
    ...);

Of course you have to cater for endianness. Boost Endian has you covered in several ways. Compare the performance trade-offs here: https://www.boost.org/doc/libs/1_80_0/libs/endian/doc/html/endian.html#overview_performance

The simplest idea here is to use the drop-in (un)aligned arithmetic types:

std::array<boost::endian::big_int32_t, 45> data;

boost::asio::async_write(socket_,
    boost::asio::buffer(data),
    ...);

To make the example a bit more life-like:

struct Message {
    boost::endian::big_uint32_t magic = 0xCAFEBABE;
    boost::endian::big_uint32_t length = 0;
    std::vector<boost::endian::big_uint32_t> data;

    std::array<boost::asio::const_buffer, 3> as_buffers() const {
         length = data.length(); // update leading length header field
         return {
               boost::asio::buffer(&magic, sizeof(magic)),
               boost::asio::buffer(&length, sizeof(length)),
               boost::asio::buffer(data),
         };
    }
};

Now you can still:

 Message message;
 message.data = {1,2,3,4,5,6,7,8};

boost::asio::async_write(socket_,
    message.as_buffers(),
    ...);