Few questions on Serialization/Deserialization

Question

A few years ago I wrote a small Stock trading TCP App while learning Socket and at that time I didn't know about Json or binary serialization/deserialization and following along some articles/reply from this site or somewhere, I made a Struct like this for transmission between clients and server:

[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct TradeItem
{
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 15)]
    public string Status;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 20)]
    public string EndPoint;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 10)]
    public string ItemName;      
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 5)]
    public string Mode;
    public int Quantity;
    public double Price;
    public int PartyCode;
    public int OrderNo;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 20)]
    public string BrokerBought;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 20)]
    public string BrokerSold;
    public int PartySold;
    public int PartyBought;
    public int QtyTraded;
    public double PriceTraded;
}

and for converting it to byte[] and reverting to Struct, I'd these helper methods:

public byte[] Pack(TradeItem ti)
{
    int size = Marshal.SizeOf(ti);
    byte[] array = new byte[size];
    IntPtr ptr = Marshal.AllocHGlobal(size);
    Marshal.StructureToPtr(ti, ptr, false);
    Marshal.Copy(ptr, array, 0, size);
    Marshal.FreeHGlobal(ptr);
    return array;
}

public TradeItem UnPack(byte[] arr)
{
    TradeItem ti = new TradeItem();
    int size = Marshal.SizeOf(ti);
    IntPtr ptr = Marshal.AllocHGlobal(size);
    Marshal.Copy(arr, 0, ptr, size);
    ti = (TradeItem)Marshal.PtrToStructure(ptr, ti.GetType());
    Marshal.FreeHGlobal(ptr);
    return ti;
}

with all these I'd to send one item at a time! I still don't know how to deal with arbitrary size of List<TradeItem >! If someone knows how to deal with List<TradeItem >, It'd be nice to know that as well. Now I can send back and forth a single TradeItem and/or List<TradeItem> with Newtonsoft.Json or BinaryFormatter. Is there any benefit of using those kind of Marshaling/Cryptic Attributes over easy Serialization/Deserialization with Newtonsoft.Json or BinaryFormatter? Over my Local Network I always receive what I send with these:

//send
var data = new List<TradeItem>;
.
.
.
byte[] array = null;
using(var stream = new MemoryStream())
{
    var formatter = new BinaryFormatter();
    formatter.Serialize(stream, data);
    array = stream.ToArray();           
}

//receive
var formatter = new BinaryFormatter();
var list = formatter.Deserialize(new MemoryStream(e.Buffer)) as List<TradeItem>;

or

//send
var array = Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(data));

//receive
var json = Encoding.UTF8.GetString(e.Buffer);
var list = JsonConvert.DeserializeObject<List<TradeItem>>(json);

I just have to create a big enough byte[]! Am I guaranteed to get what I send using these BinaryFormatter and JsonConvert provided that I've a big enough byte[]?

The second option(`var array = Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(data)):`) seems better :) — Momoro, Oct 26 '19 at 06:31
Take a look at [What are the deficiencies of the built-in BinaryFormatter based .Net serialization?](https://stackoverflow.com/q/703073/3744182). Also note that since `BinaryFormatter` streams contain embedded type information, `BinaryFormatter` is extremely vulnerable to attack type injection attacks detailed [here](https://www.blackhat.com/docs/us-17/thursday/us-17-Munoz-Friday-The-13th-JSON-Attacks-wp.pdf) and [here](https://stackoverflow.com/q/39565954). — dbc, Oct 26 '19 at 06:44
One method if you are sending 10 TradeItems send the number of items. Then send before the each TradeItem the size of the TradeItem. Any data you send the receive end need to know where each object terminates. So when you send binary you usually include a byte count before the object. — jdweng, Oct 26 '19 at 10:57
@jdweng, that's where it becomes complicated! First I've to read first 4 bytes to determine the size of array, then create an array using that determined size, then reading `while < size`, I did that a few times earlier. Is it done by `Json.Convert/BinaryFormatter` automatically? — , Oct 26 '19 at 23:50
Yes. When sending data any type data, the data must be packed so at the receive end the data can be unpacked so you get the original data unchanged. — jdweng, Oct 27 '19 at 10:20
@jdweng, great, now I can safely get rid of packing data manually with size in the first 4 bytes! — , Oct 27 '19 at 10:27
You are sending binary data and need a method of determining the end of the binary data. You can eliminate the size between the indexes in the array but the array still needs a size or EOF. — jdweng, Oct 27 '19 at 11:03
@jdweng, My `byte[]` was 512 and I used to send 4/5 objects, with a `string` and `int`, at a time while testing so I always received full list in a single shot. After setting it to `byte[32]` I got error when sending more than 2 objects at a time! Instead of `` I've checked for `]` in a while loop and used a `StringBuilder` to store the data and used that string in `JsonConvert.DeserializeObject` call! I've edited my post [here](https://stackoverflow.com/questions/58576316/implementing-server-with-socketasynceventargs) to show how have I done so. — , Oct 27 '19 at 14:59
TCP is not guaranteed to send data in one block. Windows uses timers and it is possible for the timers to break a message into multiple blocks. There are a lot of other cases where TCP get broken into pieces. TCP use datagrams that have max size of ~1500 bytes. The specification allow different software vendor to use different size that is why I said approximately 1500 bytes. Then TCP can be split into pieces when Port Forwarding is used to go through a firewall. Why are you using a string builder for binary data it will corrupt the data? Use a List instead. — jdweng, Oct 27 '19 at 21:28
@jdweng, I've another question [here](https://stackoverflow.com/questions/58588316/reading-byte-in-client-server-tcp-app-in-a-single-call) relating to the size of packet. — , Oct 28 '19 at 09:19

Few questions on Serialization/Deserialization

0 Answers0