15

I've been using WebServices at moving data across the wire and that has served me pretty well. It excels at sending small pieces of data. As soon as you have to move deep object trees with lots of properties, the resulting XML soup takes 100k of data and turns it into a 1MB.

So I've tried IIS Compression, but it left me underwhelmed. It compressed data well, but the trade off was in compression/decompression. Then I've serialized the objects via BinaryFormatter and sent that across. This was better, however, speed of encode/decode still remains.

Anyway, I am hearing that I am stuck in the 00s and now there are better ways to send data across the wire such as ProtocolBuffers, MessagePack, etc...

Can someone tell me whether these new protocols will be better suited for sending large pieces of data and whether I am missing some other efficient ways to do this?

By efficient, I mean amount of bandwidth, speed of encode/decode, speed of implementation, etc...

AngryHacker
  • 59,598
  • 102
  • 325
  • 594
  • 1
    Are you communicating within an intranet or publicly via the internet? – Jordan Parmer Jun 27 '11 at 19:15
  • @j0rd4n Intranet, but it's spread across the country via different providers and most of the time, it's much slower than the internet. – AngryHacker Jun 27 '11 at 20:00
  • Here's [some interesting numbers](http://www.servicestack.net/benchmarks/NorthwindDatabaseRowsSerialization.1000000-times.2010-02-06.html), which favor protobuf-net (or any equivalent protocol buffers implementation - but conveniently, protobuf-net tends to be pretty easy to retrofit to an existing model). It ties in pretty much to every metric I've done on this, and I have done many. – Marc Gravell Jun 27 '11 at 20:03

4 Answers4

11

It depends on what's making up the bulk of your data. If you've just got lots of objects with a few fields, and it's really the cruft which is "expanding" them, then other formats like Protocol Buffers can make a huge difference. I haven't used MessagePack or Thrift, but I would expect they could have broadly similar size gains.

In terms of speed of encoding and decoding, I believe that both Marc Gravell's implementation of Protocol Buffers and my own will outperform any of the built-in serialization schemes.

Jon Skeet
  • 1,421,763
  • 867
  • 9,128
  • 9,194
  • actually, I'm always more interested in bandwidth - we win there too ;p As a side note, I *have* looked at MsgPack due to some silly "4 times faster than protocol buffers" claim, which of course was absurdly untrue ;p – Marc Gravell Jun 27 '11 at 20:04
  • @Marc Does it make sense to further compress the proto-buffed objects using IIS Compression? Or will that just unnecessarily increase CPU utilization for little in return? – AngryHacker Jun 27 '11 at 21:18
  • @AngryHacker: I wouldn't like to say - but you might as well try it. – Jon Skeet Jun 27 '11 at 22:29
  • 1
    @AngryHacker: To expand on Jon's message, Protocol buffers encodes data with realtively little overhead, but it does not include tarditional compression. Therefore, depending on your data the results may benefit from traditional compression, but that could vary, and compression for the wire always involves a tradeoff between latency and "bandwidth", so whether IIS compression would be a benefit or not depends on your application. – Kevin Cathcart Jun 28 '11 at 21:32
6

This depends heavily on where your priorities lie, and what type of client you're using.

WCF provides some great ways to push data across the wire, including many binding options as well as rather efficient serializers such as the DataContractSerializer. That being said, many of these things require a "rich" client that's also using WCF.

If that's not an option, then something like Protocol Buffers may be a very good approach. This provides a very fast serialization/deserialization as well as a reasonable transmission size for most data.

Reed Copsey
  • 554,122
  • 78
  • 1,158
  • 1,373
  • Here's an interesting post re different binding options: http://stackoverflow.com/questions/3790728/performance-tests-of-serializations-used-by-wcf-bindings – Marc Gravell Jun 27 '11 at 20:08
2

Have you checked out Protobuf-net ??

BentOnCoding
  • 27,307
  • 14
  • 64
  • 92
1

Using Simon Hewitt's library (written in C#) to perform the serialisation it is both efficient in terms of bandwidth and speed of encoding/decoding. The library is distributed as C# source code.

The only gotcha is to avoid .NET serialisation to kick in - for data structures not supported by the library explicit encoding must be done in client code to avoid .NET serialisation to be invoked. This affects the speed of implementation, but I got an improvement of a factor of 2-3 for the size and a factor of 20-40 times for the time taken for the serialisation.

Community
  • 1
  • 1
Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131