How to ensure correct data is received, if serial receiver execution time varies

Question

I have a serial communication between STM32 and Linux computer. STM transmits every 10ms 6 bytes of data "iiiiiC" (i = transmission_counter % 10, C = character 'C'). The receiver however does not have a specific iteration time and may read serial faster or slower than 10ms. Also, the read function should be non-blocking.

This means that sometimes received data:

RD = '111'
RD = '11C2'
RD = '222'
RD = '2C33'

I want usable_data = 22222C

and sometimes:

RD = '333C44444C'
RD = '55555C66666'
RD = 'C77777C8888'

I want usable_data = 77777C

Code is similar to following:

int main() {

  int serial_port = open("/dev/ttyACM1", O_RDWR);
  struct termios tty;

  // Read in existing settings, and handle any error
  if(tcgetattr(serial_port, &tty) != 0) {
      printf("Error %i from tcgetattr: %s\n", errno, strerror(errno));
      return 1;
  }

  tty.c_cflag &= ~PARENB; // Clear parity bit, disabling parity (most common)
  tty.c_cflag &= ~CSTOPB; // Clear stop field, only one stop bit used in communication (most common)
  tty.c_cflag &= ~CSIZE; // Clear all bits that set the data size 
  tty.c_cflag |= CS8; // 8 bits per byte (most common)
  tty.c_cflag &= ~CRTSCTS; // Disable RTS/CTS hardware flow control (most common)
  tty.c_cflag |= CREAD | CLOCAL; // Turn on READ & ignore ctrl lines (CLOCAL = 1)

  tty.c_lflag &= ~ICANON;
  tty.c_lflag &= ~ECHO; // Disable echo
  tty.c_lflag &= ~ECHOE; // Disable erasure
  tty.c_lflag &= ~ECHONL; // Disable new-line echo
  tty.c_lflag &= ~ISIG; // Disable interpretation of INTR, QUIT and SUSP
  tty.c_iflag &= ~(IXON | IXOFF | IXANY); // Turn off s/w flow ctrl
  tty.c_iflag &= ~(IGNBRK|BRKINT|PARMRK|ISTRIP|INLCR|IGNCR|ICRNL); // Disable any special handling of received bytes

  tty.c_oflag &= ~OPOST; // Prevent special interpretation of output bytes (e.g. newline chars)
  tty.c_oflag &= ~ONLCR; // Prevent conversion of newline to carriage return/line feed
  // tty.c_oflag &= ~OXTABS; // Prevent conversion of tabs to spaces (NOT PRESENT ON LINUX)
  // tty.c_oflag &= ~ONOEOT; // Prevent removal of C-d chars (0x004) in output (NOT PRESENT ON LINUX)

  tty.c_cc[VTIME] = 0;    // No blocking, return immediately with what is available
  tty.c_cc[VMIN] = 0;

  cfsetispeed(&tty, B115200);
  cfsetospeed(&tty, B115200);

  // Save tty settings, also checking for error
  if (tcsetattr(serial_port, TCSANOW, &tty) != 0) {
      printf("Error %i from tcsetattr: %s\n", errno, strerror(errno));
      return 1;
  }
  
while (1){

    char read_buf [100]; // Large serial buffer so even if the receiver is slow the most recent data will be in this array

    memset(&read_buf, '\0', sizeof(read_buf));

    int num_bytes = read(serial_port, &read_buf, sizeof(read_buf));

    std::cout << num_bytes << "  " << read_buf << std::endl;;

    ... unknown time ahead ...
  }

  close(serial_port);
  return 0; // success
}

I tried some workarounds, but they don't solve all the problems or I'm otherwise not happy with them:

// For cases where receiver is slow and receives more than one complete message at once
char part [6];    
uint16_t index = 0;
    for (index; index < 6; index++){
      if (read_buf[index] == 'C'){
        while (read_buf[index] == 'C'){
          index += 6;
        }
        index -= 6;
        break;
      }
    }    
memcpy(part, read_buf + index - 5, 6);

.

// For cases where received data does not fit into read_buffer, i.e. we are reading old data - flush it.
int num_bytes = read(serial_port, &read_buf, sizeof(read_buf));
while (num_bytes == sizeof(read_buf)){
  num_bytes = read(serial_port, &read_buf, sizeof(read_buf));
}

.

// For cases where received data is out of phase (message does not end with a 'C'). Reads one character at a time until message ends with 'C'. Has many problems.
if (read_buf[sizeof(read_buf)-1] != 'C'){
  read(serial_port, &read_buf, 1);
}

Answer 1

but since usable_data might be split into parts (part of usable_data in the current message, part of it in the previous one) it's difficult to merge the correct bytes together.

The best way to handle that specific challenge is to let read() take care of that for you by leveraging the fact that You don't have to point the second argument of read() to the start of the buffer. You can make it write the data from wherever it last stopped.

constexpr std::size_t msg_len = 6;
std::array<char, msg_len> read_buf;
int received = 0;

void poll_port() {
  auto count = read(
     serial_port, 
     read_buf.data() + received, // Append to whatever we already have
     msg_len - received);        // Only read data up to the end of the buffer.

  if(count != -1) {
    received += count;
    if(received == msg_len) {
      handle_message(read_buf);
      received = 0;
    }
  }
  else if(errno != EAGAIN) {
    // we failed for some other reason than lack of data.
  }
}

Combine this with configuring the port in non-buffered mode as per https://stackoverflow.com/a/5616108/4442671, to make read() non-blocking, and you'll have pretty much what you need.

Obviously, this is a simplified example that assumes all messages are of the same length and that no data corruption happens along the way, but it should set you on the general path.

Answer 2

How to ensure correct data is received, if serial receiver execution time varies

Write a parser that will parse the incoming data and accumulate them in a buffer and notify "upper processing stage" only when a full packet is received.

bytes of data "iiiiiC"

That's trivially trivial to write, a sample parser might look like the following:

struct myrecv {
   std::array<char, 6> buf;
   unsigned pos; 
   // pointer to a function that return EOF or valid character
   std::function<int()> readc;
   // pointer to callback function to call when full buffer is received
   std::function<void(std::array<char, 6>&)> cb;

   myrecv(std::function<int()> readc, 
        std::function<void(std::array<char, 6>&)> cb) :
        readc(readc), cb(cb), pos(0) {}

   int process() {
       buf[pos] = readc();
       if (buf[pos] == EOF) return ETIMEDOUT;
       pos++;
       if (pos != buf.size()) {
          return 0;
       }
       pos = 0;

       // check message integrity
       if (buf.end() != 'C') {
          return -EINVAL;
       }

       cb(buf);
       return 1;
   }
};