16

I'm looking into a mechanism for serialize data to be passed over a socket or shared-memory in a language-independent mechanism. I'm reluctant to use XML since this data is going to be very structured, and encoding/decoding speed is vital. Having a good C API that's liberally licensed is important, but ideally there should be support for a ton of other languages. I've looked at google's protocol buffers and ASN.1. Am I on the right track? Is there something better? Should I just implement my own packed structure and not look for some standard?

bmdhacks
  • 15,841
  • 8
  • 34
  • 55
  • might want to see [performant-entity-serialization-bson-vs-messagepack-vs-json](http://stackoverflow.com/questions/6355497/performant-entity-serialization-bson-vs-messagepack-vs-json) as well. – nawfal Jul 17 '14 at 07:34

10 Answers10

11

Given your requirements, I would go with Google Protocol Buffers. It sounds like it's ideally suited to your application.

Greg Hewgill
  • 951,095
  • 183
  • 1,149
  • 1,285
  • 1
    The problem I have with Protocol Buffers is that the C interface is positively awful. I'm leaning towards asn1c at the moment. – bmdhacks Sep 29 '08 at 14:28
  • 1
    Five years later, there are a number of good C interfaces for protocol buffers, so I'm marking this as the correct answer. – bmdhacks Mar 14 '13 at 20:25
3

You could consider XDR. It has an RFC. I've used it and never had any performance problems with it. It was used in ONC RPC and has an and comes with a tool called rpcgen. It is also easy to create a generator yourself when you just want to serialize data (which is what I ended up doing for portability reasons, took me half a day). There is an open source C implementation, but it can already be in a system library, so you wouldn't need the sources.

ASN.1 always seemed a bit baroque to me, but depending on your actual needs might be more appropriate, since there are some limitations to XDR.

Community
  • 1
  • 1
Hans van Eck
  • 390
  • 1
  • 4
3

Just wanted to throw in ASN.1 into this mix. ASN.1 is a format standard, but there's libraries for most languages, and the C interface via asn1c is much cleaner than the C interface for protocol buffers.

bmdhacks
  • 15,841
  • 8
  • 34
  • 55
  • Erlang has a superb support for ASN.1! It is very easy to use, and it is really well-documented. Read more [here](https://erlang.org/doc/apps/asn1/) ([PDF](https://erlang.org/doc/apps/asn1/asn1.pdf)). Documentation for the asn1ct module can be found [here](https://erlang.org/doc/man/asn1ct.html). – odiferousmint Apr 25 '21 at 23:44
2

JSON is really my favorite for this kind of stuff. I have no prior experience with binary stuff in it though. Please post your results if you are planning on using JSON!

SchizoDuckie
  • 9,353
  • 6
  • 33
  • 40
  • I'm unfortunately going to have to rule JSON out due to the parsing overhead. I'm really looking for formats that store the data in binary that does not have to be parsed. – bmdhacks Sep 27 '08 at 23:48
  • You may be overestimating overhead -- well-written JSON parser is pretty darn efficient; and not all binary parser are well written (meaning sometimes they are slower). So I would first measure performance of alternatives instead of assuming you know the answer. – StaxMan Feb 06 '11 at 06:03
  • Update from my side: I've just done JSON with binary content, by manually base64 encoding the binary blobs. It works pretty well! – SchizoDuckie Oct 06 '12 at 18:36
2

Thrift is a binary format created by Facebook. Here's a comparison with google protocol buffers.

bmdhacks
  • 15,841
  • 8
  • 34
  • 55
1

Check out Hessian

Mauricio Scheffer
  • 98,863
  • 23
  • 192
  • 275
1

There is also Binary XML but it seems not stabilized yet. The article I link to gives a bunch of links which might be of interest.

PhiLho
  • 40,535
  • 6
  • 96
  • 134
1

Another option is SNAC/TLV which is used by AOL in it's Oscar/AIM protocol.

bmdhacks
  • 15,841
  • 8
  • 34
  • 55
1

Also check out Muscle. While it does quite a bit, it serializes to a binary format.

Joel Lucsy
  • 8,520
  • 1
  • 29
  • 35
0

Few Thing's you need to Consider

1. Storage
2. Encoding Style (1 byte 2 byte)
3. TLV standards

ASN.1 Parser is the good for binary represenations the best part is ASN.1 is a well-established technology that is widely used both within ITU-T and outside of it. The notation is supported by a number of software vendors.

anish
  • 6,884
  • 13
  • 74
  • 140