How to simply serialize complex structures and send them over a network in Qt

Question

I am creating a client server application and now I am dealing with a most simple way to get some of the classes serialized, delivered to other side, and put back into class that I can work with later.

I realize this is not simple, some might say impossible in low level languages like C or C++, but it actually is doable with lot of coding. I am wondering if someone else already didn't create a solution for this, which is portable and works so that I don't need to reinvent a wheel.

Currently my solution (maybe a bit too complex):

Every class that is meant to be serialized and deserialized is inherited from abstract class Serializable which contains 2 functions:

QHash<QString, QVariant> ToHash();
void LoadHash(QHash<QString, QVariant> hash);

First function creates a QHash which contains all public and private variables. Second function loads this hash and fills the values where they belong. So that I can turn every structure I need to serialize to QHash and back as I need.

Now the tricky part is to deliver this to other side over network. I decided to create a super simple TCP protocol for this.

I can use QDataStream in order to convert this QHash into a byte array and (the documentation says that, although I am afraid there might be some hidden caveat) back into QHash using the same class.

That means, I have a way to serialize structures to QByteArray's and I can convert them back, now I need to send and receive them.

Because I am transferring binary data I decided not to use some sort of separator for these "data blocks" because I don't want to mess up with complicated escaping of these separators, instead I decided to create a fixed-size header (I store byte size in constant called HEADER_SIZE, currently it's 8 bytes). The size of header must be same on client and server.

What I am going to do is that I take the generated QByteArray (serialized class), get its size and create header from it which is long exactly HEADER_SIZE bytes, put a real size of serialized structure in there and prepend it to the QByteArray. Then I send this array to other side.

On other side I am waiting for first HEADER_SIZE bytes and then get exactly as many bytes as described in header, forming the QHash back from it. (I don't have this code yet, it's just a theoretical algorithm right now).

Is this a right way to do this? Isn't there more simple way? Or something that already handles this problem for me? Are there some caveats I should be aware of?

Just to make it a little bit more clear, here is this pseudo diagram of what I am trying to do:

Answer 1

You can and should leverage QDataStream for that. Instead from deriving from Serializable, simply implement QDataStream & operator<<(QDataStream &, Type & const) and QDataStream & operator>>(QDataStream &, Type &) for each Type you want to serialize.

You'd then simply dump all data to a QByteArray via a QDataStream. You'd transmit the size of the array, followed by its contents. On the receiving end, you receive the size, then the contents, set a datastream on it, and pull the data out. It should be seamless.

The data block separation is handled automatically for you as long as the individual streaming operators are implemented correctly. The choice of a QHash as a means of serialization is unnecessarily limiting - it may be a good choice for some classes, but not for others.

Instead of serializing into a QHash, you should be serializing into a QDataStream. But the operators for that can be free-standing functions, so you don't need to derive from any special interface class for that. The compiler will remind you of any missing operators.

This is a very simple example, working over UDP.

This is a larger example that shows the details of future-proof versioning, and shows serialization of a rather complex data structure - a QAbstractItemModel.

Generally speaking, the serialization of, say, three objects a,b,c might look as follows:

static const QDataStream::Version kDSVersion = QDataStream::Qt_5_5;

void Foo::send() {
  QByteArray buf;
  QDataStream bds(&buf, QIODevice::WriteOnly));
  bds.setVersion(kDSVersion);
  bds << a << b << c;

  QDataStream ds(socket, QIODevice::WriteOnly);
  ds.setVersion(kDSVersion);
  ds << buf; // buffer size followed by data
}

On the receiving end:

void Foo::readyReadSlot() {
  typedef quint32 QBALength;
  if (socket->bytesAvailable() < sizeof(QBALength)) return;
  auto buf = socket->peek(sizeof(QBALength));
  QDataStream sds(&buf);
  // We use a documented implementation detail:
  // See http://doc.qt.io/qt-5/datastreamformat.html
  // A QByteArray is serialized as a quint32 size followed by raw data.
  QBALength size;
  sds >> size;
  if (size == 0xFFFFFFFF) {
    // null QByteArray, discard
    socket.read(sizeof(QBALength));
    return;
  }
  if (socket->bytesAvailable() < size)  return;
  QByteArray buf;
  QDataStream bds(&socket);
  bds.setVersion(kDSVersion);
  bds >> buf;
  QDataStream ds(&buf);
  ds.setVersion(kDSVersion);
  ds >> a >> b >> c;
}

Answer 2

Yes, someone already invented this.

Suggest you have a look at google protocol buffers. Not only is it portable between c++ implementations and processor architectures, it's also portable between different languages.

https://developers.google.com/protocol-buffers/

Answer 3

You can use JSON protocol,

Send structures like this:

{"employees":[
{"firstName":"John", "lastName":"Doe"},
{"firstName":"Anna", "lastName":"Smith"},
{"firstName":"Peter", "lastName":"Jones"}]}

You can make it as complex as you want.

Qt has QJsonObject library that makes very easy to work with it.