How to avoid cross language dependency in serialization/deserialization using TCP?

Question

I need to create TCP chat with C++ clients and Python server(already started), I have messages in c++ class like

class Message{
public:
uint64 utc_time;
uint64 token;
string content;
};

I am sending this from client to server, on server I have priority queue by utc_time and need to broadcast to others. My question is how to serialize this, which format to use so to avoid any cross language dependencies on size type size ? (maybe is going to be more meta data in future so need kind a little generic) ? Can anyone give me advice which format to use for serialization(or to flush only like bytes) ?

class Persistent:
public:
    Persistent(int sz):objSize(sz){}
    void write(std::ostream& out)const{out.write((char*)this, objSize);}
    void read(std::istream& in){in.read((char*)this, objSize);}
private:
    int objSize;
};

I thought of other possibility to have deserializator in c++ on server and call from python if that is possible. Any elegant solution to this problem ?

Answer 1

If you really want to go cross language and cross platform without having to worry about where the message ends, have a look at the combination of Google Protobuf and ZeroMQ.

When using regular sockets you would first read the size of the message (you would prepend this) and then you would know from where to where the byte-array is a complete message.

Example protobuf + zmq usage:

message Message {
    optional uint64 utc_time = 1;
    required uint64 token = 2;
    optional string content = 3;
}

Use the protobuf compiler to generate C++ code (or ruby/python/etc).

To use it in your code:

#include <Message.pb.h>

Message msg;
msg.set_token(1);
msg.set_content("Hello world");

To send it using zmq:

std::string serialized = msg.SerializeAsString();
zmq::message_t reply(serialized.size());
memcpy(reply.data(), serialized.data(), serialized.size());
zmq_socket.send(reply);

To receive it using zmq:

zmq::message_t request;
zmq_socket.recv(&request); // blocking
Message recv_msg;
recv_msg.ParseFromArray(request.data(), request.size());

Answer 2

Using ZeroMQ is a good start since it takes care of all the transport work for you. The best way to serialize depends on the kind of work you're doing. Since you're doing a chat application, efficiency is not a concern, so I'd use a self-describing text format, which is simplest to debug, trace, log, and use. Anything like protobufs or msgpack is going to be extra work for no measurable reward. You can use XML, JSON, HTTP-style headers, name=value pairs, etc.

When you get to making really high-volume messages (hundreds of thousands per second) or really large ones (over say 1K bytes), then you can start to think of different ways to reduce the message size. I'd personally recommend sticking with the simplest possible text format until you do have a performance issue, then switching to the most efficient binary format for the cases that need it. But not in a chat app...

Answer 3

I like to use JSON, assuming you have a good buffered interface to the socket and a JSON parser based on streams. The nice thing about JSON is you don't need to specify the length of each message. A properly written JSON parser can tell when it hits the end of the "object". So your reader object can just parse the JSON as it comes over the wire and when you reach the end of the initial object, return it to the system as one message.

If JSON is overkill for your data, there's always plain text. Most of the internet runs on plain text (POP, IMAP, HTTP, FTP, etc.). And that's because plain text is the easiest thing to using cross-platform/cross-language.