C# Serial Port Buffer Looking for Specific Binary Patterns

Question

I have spent a lot of team researching the proper ways to use the serial port in C# such that you don't have problems reading in data. I think I have a solution which is pretty close to functional, but I have some glitches every once in a while which I cannot seem to figure out.

My goal: Read formatted binary messages from the serial port, and pass them along to a processor.

The message format looks something like this:

(MSG-HEADER)(MSG-ID)(MSG-LENGTH)(DATA0)(DATA1)(DATA2)...(DATA-N)

Each "word" in the data is 16 bits (2-bytes). My basic approach is to start in a "read message header" state, where each time the serial data received event occurs, I read from the serial port, store the data in a buffer, and then check to see if I detect the message header. If I detect the message header, I move into a "read data" state, where I keep reading data into a data buffer until I have read bytes.

This seems to work pretty well, except occasionally I see "data glitches". Where I end up storing a message that looks something like this:

(MSG1-HEADER)(MSG1-ID)(MSG1-LENGTH)(DATA0)(DATA1)(DATA2)(MSG2-HEADER)(MSG2-ID)..etc

Basically, every so often I get a proper message header, message ID, message length, then the data starts (typically around 200 bytes), and right in the middle of that data I see another message header, message id, and message length, and presumably the start of another message data section. And I can't seem to figure out why.

Here is the code in the serial port data received event I am using:

    public byte[] headerBuff = new byte[500];
    public byte[] dataBuff = new byte[500];
    public byte[] tempBuff = new byte[500];
    public int bytesRead;
    public int dataPos;
    public int dataMsgLen;
    public int dataBytesRead = 0;
    public bool READ_HEADER = true;
    ConcurrentQueue<byte[]> serialQ = new ConcurrentQueue<byte[]>();


   //private void si_DataReceived(byte[] data)
    private void si_DataReceived(object s, EventArgs e)
    {
        //If we're supposed to be reading the header, read some bytes and look
        // For the header identification sequence (0xF989)
        if (READ_HEADER)
        {
            //Read some bytes, save how many we read
            bytesRead = comport.Read(headerBuff, 0, comport.BytesToRead);

            //Any time we call comport.REad, we automatically log those bytes to a file
            using (BinaryWriter writer = new BinaryWriter(File.Open(defDataDir, FileMode.Append)))
                writer.Write(headerBuff.Skip(0).Take(bytesRead).ToArray());

            //Loop through the bytes we just read and look for sequence
            for (int i = 0; i < (bytesRead-1); i++)
            {
                if (headerBuff[i] == 0xF9 && headerBuff[i + 1] == 0x89)
                {
                    //We have identified a header
                    // Lets copy it into a new array
                    dataPos = bytesRead-i;
                    Array.Copy(headerBuff, i, dataBuff, 0, dataPos);
                    dataMsgLen = dataBuff[4];

                    //Now we can switch to message logging
                    READ_HEADER = !READ_HEADER;
                    Array.Clear(headerBuff, 0, headerBuff.Length); //clear the buffer for next time
                    break; // don't need to look for headers anymore
                }
            }
        }

        //If we are done reading the header, let's wait until we get 
        // enough bytes to store the data message
        else if (!READ_HEADER)
        {
            // Read some bytes into temp array
            var tempNumBytes = comport.Read(tempBuff, 0, comport.BytesToRead);

            //ADD this into data buffer
            Array.Copy(tempBuff, 0, dataBuff, dataPos + dataBytesRead, tempNumBytes);

            //Increment our counter
            dataBytesRead += tempNumBytes;

            //Save to stream
            using (BinaryWriter writer = new BinaryWriter(File.Open(defDataDir, FileMode.Append)))
                writer.Write(tempBuff.Skip(0).Take(tempNumBytes).ToArray());

            //Add to FIFO if we have read enough bytes
            if (dataBytesRead >= (dataMsgLen * 2))
            {
                //Debug.Print(BitConverter.ToString(dataBuff));
                serialQ.Enqueue(dataBuff.Select(x => x).ToArray()); // Add to queue for processing
                READ_HEADER = !READ_HEADER; // Go back to looking for headers
                dataBytesRead = 0;
            }
        }
    }

I appreciate any help, let me know if you need any clarifications.

Thank you in advance.

Answer 1

All, Thank you for your comments. Based on what I read, I re-wrote the serial data handler (see code below) and it seems to be working much better. I have had it running for about ten minutes now and I haven't seen this glitch at all.

    //Declare some public variables for serial port reading
    public byte[] headerBuff = new byte[500];
    public byte[] dataBuff = new byte[500];
    public byte[] tempBuff = new byte[500];
    public int headerBytesRead = 0;
    public int dataBytesRead = 0;
    public const int HEADER_LENGTH = 10;
    public int dataInd;
    public int fullMsgLen;
    public byte[] queuePop;

    //Declare some states
    public bool READ_HEADER = true;

    //Where will we store the data log?
    public string defDataDir;

    //Declare a public queue as a FIFO for incoming serial data once the 
    // buffer is full
    ConcurrentQueue<byte[]> serialQ = new ConcurrentQueue<byte[]>();


   //private void si_DataReceived(byte[] data)
    private void si_DataReceived(object s, EventArgs e)
    {
        //If we're supposed to read the headers, do that
        if(READ_HEADER)
        {
            //Read some bytes
            var numBytesRead = comport.Read(tempBuff, 0, comport.BytesToRead);

            //Any time we call comport.Read, we automatically log those bytes to a file
            using (BinaryWriter writer = new BinaryWriter(File.Open(defDataDir, FileMode.Append)))
                writer.Write(tempBuff.Skip(0).Take(numBytesRead).ToArray());

            //Add these bytes to a header array
            Array.Copy(tempBuff, 0, headerBuff, headerBytesRead, numBytesRead);

            //Increment headerBytesRead counter
            headerBytesRead += numBytesRead;

            //Loop through header and see if we have a header
            if(headerBytesRead>=HEADER_LENGTH)
            {
                //Loop through all the header bytes read so far
                for(int i=0; i<headerBytesRead;i++)
                {
                    //Look for the header start word.  Note, 3rd bool statement
                    // here is to make sure we have enough bytes left to identify a header
                    // e.g. read 12 bytes, and bytes 11 and 12 are 0xF9 and 0x89, then we 
                    // clearly don't have the rest of the header (since it is length 10)
                    if(headerBuff[i]==0xF9 && headerBuff[i+1]==0x89 && (headerBytesRead-i-1)>=9)
                    {
                        //We have identified a header, and have enough following characters to save it
                        //Copy the header into the data array
                        Array.Copy(headerBuff, i, dataBuff, 0, headerBytesRead - i);
                        dataInd = headerBytesRead - i;

                        //Save the message length
                        fullMsgLen = dataBuff[4]*2 + HEADER_LENGTH;

                        //Switch over to reading data
                        READ_HEADER = !READ_HEADER;

                        //Reset our header length counter
                        headerBytesRead = 0;

                        //Clear the header buffer for next time
                        Array.Clear(headerBuff, 0, headerBuff.Length); 
                        break; // don't need to look for headers anymore
                    }
                }
            }
        }

        //Handle reading data into buffer here
        else if (!READ_HEADER)
        {
            //We've just been told to start reading data bytes, and we know how many
            var numBytesRead = comport.Read(tempBuff, 0, comport.BytesToRead);

            //Any time we call comport.Read, we automatically log those bytes to a file
            using (BinaryWriter writer = new BinaryWriter(File.Open(defDataDir, FileMode.Append)))
                writer.Write(tempBuff.Skip(0).Take(numBytesRead).ToArray());

            //Add these bytes into the data array
            Array.Copy(tempBuff, 0, dataBuff, dataInd+dataBytesRead, numBytesRead);

            //Increment our data array counter
            dataBytesRead += numBytesRead;

            //Check to see if we have saved enough
            if((dataInd+dataBytesRead) >= fullMsgLen)
            {
                //Copy the header+msg into the queue
                serialQ.Enqueue(dataBuff.Skip(0).Take(fullMsgLen).ToArray());

                //Copy the remaining bytes back into the header buffer
                Array.Copy(dataBuff, fullMsgLen, headerBuff, 0, dataInd + dataBytesRead - fullMsgLen);
                headerBytesRead = dataInd + dataBytesRead - fullMsgLen;

                //Reset data bytes read countery
                dataBytesRead = 0;

                //Switch back to looking for headers
                READ_HEADER = !READ_HEADER;
            }
        }

    }