Broken TCP messages

Question

I have a simple TCP server that communicates with some devices via GPRS. It works correctly without any problem for a long time. Nowdays there is a change in the devices (client devices, that send data) and they send lot more data than earlier and TCP message size can be 10 times bigger than earlier. Earlier it was max 1024 bytes/data message, now it can be more than 10 thousand bytes/message.

As I see - as the nature of TCP predicted - there are a lot of messages that are 'broken', so instead of a complete message only a part of that arrives, and later can come the other part etc.

I think my server - that is based on asynchronous pattern - handles this situation in a wrong way and I think the bug is in my RecieveCallBack function. I could not find to correct way to handle this situation. How can I do this?

Here is my RecieveCallBack function:

public void RecieveCallBack(IAsyncResult ar)
    {
        StateObject state = (StateObject)ar.AsyncState;

        try
        {
            Socket handler = state.WorkSocket;

            int read = handler.EndReceive(ar);

            if (read > 0)
            {
                var data = new byte[read];
                Array.Copy(state.Buffer, 0, data, 0, read);
            }
            else
            {
                if ((handler.Connected == false) || (handler.Available == 0))
                {
                    Close(state);
                    return;
                }
            }
        }
        catch (System.Net.Sockets.SocketException)
        {
            Close(state);
        }
        catch (System.Exception exc)
        {
            Debug.Assert(false, exc.Message);
            HandleSocketError(state, exc);
        }
    }

.net4/c#/vs2010

Thanks

UPDATE: Clients are connected as they can be connected, so it is up to the GSM network. They can be connected even for days and they send data at every minutes. The protocol of the client devices is different, there are different kind of devices with different kind of protocols. One of them has delimiter and CRC at the end of the message, others does not have.

have a look at this one: http://stackoverflow.com/questions/686618/tcp-client-asynchronous-socket-callback — Davide Piras, Aug 31 '11 at 12:59
@Tom: correct terminology for your "whole group of data" is a "message". Stephen Cleary is correct, the apparent solution to your problem is that you need to implement or make use of a message framing protocol. I'd suggest you consider HTTP. — MattH, Aug 31 '11 at 13:27
Could you add what is known about the message to your question please (do they have the same format, how to they end). Are messages sent on the same connection or are connections terminated between each send? — James Kyburz, Aug 31 '11 at 13:34
Thx for the replies. James, I added to my question as an update. — Tom, Aug 31 '11 at 14:09
@Tom: Every TCP/IP protocol *must* have message framing, even if it's just a simple "every message is 20 bytes" or whatever. — Stephen Cleary, Aug 31 '11 at 18:00

score 3 · Accepted Answer · edited Jan 09 '20 at 18:50

3

You need message framing. The protocol must specify how large the messages are - usually either a constant well-known size, a length prefix, or using message delimiters.

edited Jan 09 '20 at 18:50

dbc

104,963
20
228
340

answered Aug 31 '11 at 12:59

Stephen Cleary

437,863
77
675
810

This would assume changeing the way the devices send the tcp/ip packets which he can't do. – James Kyburz Aug 31 '11 at 16:43
@James: He needs to implement message framing. I'm assuming the protocol already specifies message framing which is being ignored; if the protocol does not specify message framing, then there is no solution short of changing the devices. – Stephen Cleary Aug 31 '11 at 17:57
Don't agree message framing I agree would be the solution if and only if the devices already specify it, I am assuming they don't. There are means to check if more data is on the eg socket.Available – James Kyburz Aug 31 '11 at 18:01
@James: And what would you do with the additional data??? You *must* have a way to split it up into messages (unless it's a streaming protocol, which doesn't appear to be the case). – Stephen Cleary Aug 31 '11 at 18:02
Add it to state.Buffer, he is using TCP and TCP is stream-oriented protocol – James Kyburz Aug 31 '11 at 18:21
1

"there are a lot of messages that are 'broken', so instead of a complete message only a part of that arrives, and later can come the other part" - doesn't sound like a streaming application protocol to me. – Stephen Cleary Aug 31 '11 at 18:24
Thx for all replies, all helped me, but this showd me the right way and I could help my problem! Thanks! – Tom Sep 06 '11 at 10:53

score 1 · Answer 2 · answered Aug 31 '11 at 12:58

1

The is no such thing as packets in TCP. It is a stream oriented protocol. That means when you read the socket you can get less data than you're expecting. You need to check the read size and if you got a short read then read again for the rest of the data.

For an example in C:

int read_data(int sock, int size, unsigned char *buf) {
   int bytes_read = 0, len = 0;
   while (bytes_read < size && 
         ((len = recv(sock, buf + bytes_read,size-bytes_read, 0)) > 0)) {
       bytes_read += len;
   }
   if (len == 0 || len < 0) doerror();
   return bytes_read;
}

answered Aug 31 '11 at 12:58

Robert S. Barnes

39,711
30
131
179

"The is no such thing as packets in TCP." is at best misleading, depending on what is meant by "in TCP". I would suggest you make your declaration in terms of (IP) packets, (TCP) segments and (UDP) datagrams. – MattH Aug 31 '11 at 13:21
-1 example in C when tagged with .NET and TCP has packets as it is built on top of the IP stack. – James Kyburz Aug 31 '11 at 13:23
James Kyburz: You are wrong. There is no concept of packet whatsoever in the TCP protocol at the application level. From the user / application point of view there is just a continuous stream of bytes. Thus if you want "packets" of data, then you have to implement packets at the application level by pre-pending header information such as data payload size to each transmission. There are also other methods such as using a certain data encoding and terminating bytes to indicate end of packet. One example of this is text based protocols which use newline to indicate end of transmission. – Robert S. Barnes Aug 31 '11 at 15:04
@MattH See my comment to James. – Robert S. Barnes Aug 31 '11 at 15:05
@Robert S. Barnes: I know what you're trying to say, I just think you're saying it badly. Your commented qualifier of "concept" and "at the application level" goes some way in remedying this. – MattH Aug 31 '11 at 15:23

score 1 · Answer 3 · answered Aug 31 '11 at 16:43

I am not sure how the rest of the application looks but I think the problem is with the following line :-

Array.Copy(state.Buffer, 0, data, 0, read);

If a packet is fragmented i.e the receive callback is called multiple times for the same connection the state.Buffer will be overwritten with the last received packet, this as you copy data to offset 0 in state.Buffer.

Depending on what your needs are you need to probably need more state in StateObject :) so you can append the data rather than overwrite.

Hope this hint helps.

Broken TCP messages

3 Answers3

Linked