Qt reading serial data - working code but needs to be more reliable

Question

I'm sending a few kB of data from an Arduino microcontroller to my PC running Qt.

Arduino measures the data on command from the PC and then sends the data back like this:

void loop(){

// I wait for trigger signal from PC, then begin data acquisition
// Data are acquired from a sensor, typically few thousand 16-bit values
// 1 kHz sampling rate, store data on SRAM chip

// Code below transfers data to PC
   for(unsigned int i=0;i<datalength;i++){

     // Get data from SRAM
     msb=SPI.transfer(0x00);
     lsb=SPI.transfer(0x00);        

     // Serial write
     Serial.write(msb);
     Serial.write(lsb);
     }
Serial.flush();

} // Loop over

Qt is receiving the data like this:

MainWindow::MainWindow(QWidget *parent) :
QMainWindow(parent),
ui(new Ui::MainWindow)  
{
ui->setupUi(this);

if(microcontroller_is_available){
   // open and configure serialport
    microcontroller->setPortName(microcontroller_port_name);
    microcontroller->open(QSerialPort::ReadWrite);
    microcontroller->setBaudRate(QSerialPort::Baud115200);
    microcontroller->setDataBits(QSerialPort::Data8);
    microcontroller->setParity(QSerialPort::NoParity);
    microcontroller->setStopBits(QSerialPort::OneStop);
    microcontroller->setFlowControl(QSerialPort::NoFlowControl);
    }

connect(microcontroller, &QSerialPort::readyRead, this, &MainWindow::readData);
}

void MainWindow::readData()  // Read in serial data bytes
{
    serialData += microcontroller->readAll();

    if(serialData.length()==2*datalength){ 
// Once all serial data received
// Do something, like save data to file, plot or process
}
}

Now the above code works pretty well, but once in a while (let's say once out of every few hundred acquisitions, so less than 1% of the time) not all of the data will get received by Qt and my readData function above is left hanging. I have to reset the program. So my question is: how can I make the data transfer more reliable and avoid missing bytes?

FYI: I am aware there exists an Arduino stackexchange. I'm not posting there because this seems a problem more related to Qt than Arduino.

I've never been much for pray-for-the-best protocols like this. Would you not be better off transferring the data in well -defined blocks, with a header,,checksum/CRC, tail, timoeout and ACK/NACK for each block, so that it's always possible to resend and recover from a comms fail? — Martin James, Mar 29 '18 at 21:18
If you can afford to ignore that message, try to incorporate a timeout system? — Aditya, Mar 29 '18 at 21:44
Probably yes. Do you know the name of any established protocols that I can follow instead of hacking something together myself? — MichaelT, Mar 29 '18 at 21:45
Well, it is the same stuff, but you transmit some extra data which you can put into some logic to manage transfers. It will however add complexity and data overhead, tho it is not an issue most of the time. Your data packets appear to be constant size so that will simplify things a bit. — dtech, Mar 29 '18 at 21:47
No, `datalength' is an unsigned integer. Is that actually important? — MichaelT, Mar 30 '18 at 09:19
There is a wealth of ITU.T protocols that are free, unencumbered, well documented and even have automated test suites. You could encapsulate e.g. Google protocol buffers in HDLC or a subset of X.25. The core X.25 is essentially a formalized way of maintaining transmit and receive windows and packet retransmission. Very useful stuff. I wrap protobufs in X.25 with great success. It's robust, future-proof, and has been thus far completely problem-free on the protocol end. — Kuba hasn't forgotten Monica, Mar 30 '18 at 15:26
You definitely should have a look at [this answer](https://stackoverflow.com/a/32595398/1329652) and [this answer where I simulate an Arduino environment within a Qt app](https://stackoverflow.com/a/43703784/1329652). The fusion of the two should present one approach to a workable solution. At the bare minimum, a line-oriented ASCII protocol has clean packet delimiters and you can add CRCs to it etc. You could even implement X.25 windowing on top of ASCII. I've done that too, just didn't put it on SO yet :). It's very nice to run Arduino code within the Qt app. It makes debugging a breeze. — Kuba hasn't forgotten Monica, Mar 30 '18 at 15:30

score 2 · Accepted Answer · answered Mar 29 '18 at 21:57

2

I didn't much look into it, but it seems the problem might be related to this line:

if(serialData.length()==2*datalength)

So if you got some extra data you just give up on the whole thing? It is not guaranteed that data will arrive at neatly discrete blocks after all.

You should read in the data if length is greater or equal, read in the specified length and leave the remaining data because it is part of the next block.

It would also explain why your function hangs - if you happen to exceed 2*datalength the condition is never true.

But even if you fix this, the implementation is kinda naive and not something that can be considered fullproof. There are other things that can go wrong, and you will need to have more descriptive block data so you can figure out what went wrong and how to fix it or skip errors without throwing a wrench in the gears so to speak.

answered Mar 29 '18 at 21:57

dtech

47,916
17
112
190

I agree it's very naive. I never thought about sending the data in blocks and it appears a good idea. Let's say I send 1 kB of data from the microcontroller, then a few pre-determined bytes to signal the end of that block. If Qt reads those right it requests the next 1kB block is transferred, else the previous 1kB block again. Got it, or you would do something more elaborate? – MichaelT Mar 29 '18 at 22:14
You are still theoretically sending it in blocks of datalength*2 length, but in practice you have no way to tell where a block begins or ends, no way to account for lost or corrupt data packets or stalls. The reason your code "mostly works" is pretty much luck, again there is **no guarantee** for transfer timing and granularity, at best you can assume you will get some data, and accumulate it into what hopefully will end up a valid data block. Your usage scenario is unclear, what would you rather skip on - data transfers or presumably the data acquisition you have on the mcu side. – dtech Mar 29 '18 at 22:43
My scenario was to say I send the acquired data to the PC 1 kB at a time, assuming I am acquiring more than 1 kB (typically it is something like 16 kB). If I don't read the header/signature for a block correctly then I ask for that block to be sent again. – MichaelT Mar 29 '18 at 22:57
Well, go for it and see how it works out. Keep in mind the scope of this site is about specific programming problems, so if another problem arises, you should create a new question. Questions that are too broad are usually closed. Martin James has already listed all the things it will take to make a fullproof solution. – dtech Mar 29 '18 at 23:11
The answers so far have been helpful. I will post back the findings in the original question. Thanks. – MichaelT Mar 30 '18 at 09:16

score 0 · Answer 2 · answered Mar 29 '18 at 22:19

Some thing I would suggest is wrapping your data in an envelop. Add a header character ('H' for example) and signature ('S' maybe?) every time you wanna send the data. In the receiving part check for the first and last char of your message And make sure it is what it should be. This will eliminate the noise and non-complete data pretty much.

Qt reading serial data - working code but needs to be more reliable

2 Answers2