Do I need to create separate categories for serializing Objects for transmission over TCP

Question

I've made wrappers of the TcpClient/Listener classes which include these functions to send/receive data:

Function sendBytes(ByVal stream As NetworkStream, ByVal data As Byte()) As Boolean

Function receiveBytes(ByVal stream As NetworkStream) As Byte()

They work well, but now I'm looking to send data other than just String objects and file data.

So I'm hoping to then create these functions which can send/receive anything:

Function send(ByVal stream As NetworkStream, ByVal obj As Object) As Boolean

    Return sendBytes(stream, serialize(obj))
End Function

Function receive(ByVal stream As NetworkStream) As Object

    Return deserialize(receiveBytes(stream))
End Function

(From a bit of research, including this link, I gather that protocol buffers are the best way to go in terms of compactness and speed, so I think that's what I'll use for serialize() and deserialize(), although I may initially start with the BinaryFormatter class)

Although this would be ideal, I have no idea if this would be a sensible way to carry this out. In addition to complicated objects I would have no other way of sending, I would also be passing simple objects such as Strings, Ints and even Byte() arrays which already have their own methods to convert into binary (or not even need it at all).

I have no idea — would the serializer do extra unnecessary work on those objects, especially the Byte() array which wouldn't need to be changed at all?

If so, then would it mean I should actually write the send/receive functions so that it considers cases for different types? Which means something like this:

Function send(ByVal stream As NetworkStream, ByVal obj As Object) As Boolean

    Dim data As Byte()

    Select Case obj.GetType
        Case GetType(String)

            data = Text.Encoding.ASCII.GetBytes(obj)
        Case GetType(Int32)

            data = BitConverter.ToInt32(obj)
        Case GetType(Byte())

            data = obj           
        Case ...
        ...
        Case Else
            data = serialize(obj)
    End Select

    Return sendBytes(stream, data)
End Function

and the equivalent for receive. (I would also have to add some sort of header describing the type of object being sent too so the receive function knows what to do with it, although I haven't included it in the above example code)

I've only listed some of the types which I thought might not need serialization as examples, but of course, there are more (eg. Boolean), hence the (...).

For a transport mechanism - serialize in one assembly, deserialize in another - *start* with ProtoBuf-NET. The binaryformatter will only deserialize into the same assembly-class-version unless you do some stuff to more or less trick it. — Ňɏssa Pøngjǣrdenlarp, Aug 23 '16 at 14:41
Where is your framing protocol? Why do you want to reinvent the wheel using sockets? Why don't you use complex data structures which you then generate classes for using existing, tested tooling such as Protocol Buffers (ProtoBuf)? — CodeCaster, Aug 23 '16 at 14:41
@CodeCaster Ok, first of all, what do you mean by framing protocol? — Shuri2060, Aug 23 '16 at 14:46
If you serialize a message in `Foo.EXE` and send it to `Bar.EXE` using the binary formatter, it wont deserialize. ProtoBuf-NET is ideal for this, but start thinking in terms of Types rather than Objects — Ňɏssa Pøngjǣrdenlarp, Aug 23 '16 at 14:46
TCP is a streaming protocol which has no notion of messages. So you need to frame your messages. If that sounds alien to you, you shouldn't be writing a networking application at this level. Start at a higher level, such as REST or ProtoBuf, which hides the complexity of the underlying layers. Here you're basically asking _"How to write a (de)serializer for (de)serializing any type for transportation over TCP?"_, which is too broad. — CodeCaster, Aug 23 '16 at 14:47
@CodeCaster I think I see what you mean. Yes, the messages are already framed in the original `sendBytes` and `receiveBytes` functions, although I haven't included the code. The frame appends the length of data to the start and adds a header too. — Shuri2060, Aug 23 '16 at 14:50
@CodeCaster And no, that's not what I'm asking. I'm asking if I need to consider separate cases when using serializers. I'm asking if it'd be inefficient to use the serializer to encode simple objects such as Int/String/Byte which already can be converted to binary via the `BitConverter` class. — Shuri2060, Aug 23 '16 at 14:52
Look at the performance of ProtoBuf in your own link - are you thinking you are going to improve on that? There is more to serializing than just reducing data to binary - there has to be "stuff" there to map it back to an object — Ňɏssa Pøngjǣrdenlarp, Aug 23 '16 at 14:57
@Plutonix I understand what you meant now. So even for Byte() Arrays, there is no concern of 'extra work' being done since I would've added a header anyway which would've been 'extra work' — Shuri2060, Aug 23 '16 at 15:02
I think most of the overhead is in the framing: you cant send a raw byte array without *also* including information on what that is so the other size knows how to deserialize it. Surely the people who wrote the serializers/specs/protocol (ie Google in the case of protocol buffers), are at least as smart as you and know some data needs nothing more than something to describe it. — Ňɏssa Pøngjǣrdenlarp, Aug 23 '16 at 15:09
@Plutonix Sorry to bother you again, I did include that link about ProtoBuf since I found it during my research, but I can't find where to actually download the latest version for .net. I tried https://github.com/mgravell/protobuf-net but it has no .dll files and has left me confused. Other links on google seem to be unrelated 'download' websites. Could you pass me a link? — Shuri2060, Aug 23 '16 at 16:40
https://www.nuget.org/packages/protobuf-net/ everything is on NuGet these days — Ňɏssa Pøngjǣrdenlarp, Aug 23 '16 at 16:41
The [GitHub site](https://github.com/mgravell/protobuf-net) also has a "Download Zip" option under the big green "Clone or Download" button — Ňɏssa Pøngjǣrdenlarp, Aug 23 '16 at 16:47
Sorry, I cant/dont do chat. If you cant find the package manager its at **Tools -> NuGet Pkg Mgr -> Package Console Manager ** one of the other options lets you browse and search packages. When selected, it will add the libs to your solution, just add an `Imports` statement to use them. If you have Qs about serializing just post them — Ňɏssa Pøngjǣrdenlarp, Aug 23 '16 at 17:56

score -1 · Answer 1 · edited May 23 '17 at 11:51

-1

I suggest you to use BinaryFormatter or (better) XmlSerializer to serialize data. XmlSerializer will work in most cases and you do not need to apply any attribute on top of the classes you are going to use (this is useful if you have to serialize types you don't own the code for example).

If you use XmlSerializer you should then encode the text somewhow before sending (something like Base64)

I had problems trying to use protobuf [see here], so i discourage you from using protocol buffers for this specific scenario.

edited May 23 '17 at 11:51

Community

1
1

answered Aug 23 '16 at 14:47

Mauro Sampietro

2,739
1
24
50

Ok, that's a relief. I've already got a XMLSerializer class, but was discouraged by some comments about inefficiency compared to other methods in other links I read. But nevertheless, the main question about separate cases still stands. – Shuri2060 Aug 23 '16 at 14:49
No, you don't have separate cases anymore. Use XmlSerializer and everything is a string. Then encode that string and send. – Mauro Sampietro Aug 23 '16 at 14:50
1

XmlSerializer has quite some problems in and of itself when used for network transport, not in the first place because its output is XML. And why would you then need to encode the XML as base64? Also, hundreds of thousands of people are using ProtoBuf, no need to discourage its use if one particular use case didn't work out for you. – CodeCaster Aug 23 '16 at 14:50
@sam Then wouldn't that be inefficient since I would be converting Byte arrays to a string and then back to a Byte array? – Shuri2060 Aug 23 '16 at 14:53
@QuestionAsker if you send the raw byte array you might have to define delimiters to identify and reconstruct the byte array on the other end. With a serializer this particular part can be avoided, but you have that double step. It works with any kind of data, and maybe you don't have to convert the original data to byte. – Mauro Sampietro Aug 23 '16 at 14:56
@CodeCaster Sir, I faced very complex scenarios where i simply could not get protobuf to work properly. Base64 was needed to avoid deserialization errors with XmlSerializer. CompactFramework and WinCE was involved. I'm just trying to share my experience. – Mauro Sampietro Aug 23 '16 at 15:01
@CodeCaster No need to downvote since even if you disagree with me, or if there's a better technique, nothing i said is false or manifestly unfounded. – Mauro Sampietro Aug 23 '16 at 15:07
@CodeCaster Please exaplain concrete example of problems of XmlSerializer and BinaryFormatter over a network. – Mauro Sampietro Aug 23 '16 at 15:17
The metadata overhead, for one. Versioning for another. – CodeCaster Aug 23 '16 at 15:23
Also, the `XmlSerializer` only serializes public properties and fields. What if the OP need to send an entire object including all its private fields too? – Visual Vincent Aug 23 '16 at 16:49
For future readers.. as you can read no technical problem arise from using a XmlSerializer and investigating on the big 'problems' these serializers raise we ended up with nothing. You can use it most times with no worries unless you have to launch missiles through space or drive drones remotely. This covers 99% of real world use cases. Don't overcomplicate things just because it's cool. If you absolutely need versioning go with protobuf. Don't need it just because it is available. – Mauro Sampietro Aug 24 '16 at 07:25
Even Marc Gravell the creator of protobuf-net does not disdain XmlSerializer. http://stackoverflow.com/a/4143614/711061 – Mauro Sampietro Aug 24 '16 at 07:51

Do I need to create separate categories for serializing Objects for transmission over TCP

1 Answers1