1

We have developed a system which integrates with camera and micro controller. GUI shows the image from the camera and serial count from micro controller, we used a serial thread to poll the data from micontroller and emitted the signal to GUI to display it and also we have used a separate thread to capture image and pass it to main thread. The problem with the application is when the system is in idle state, the GUI freezes and we have to restart the application to start working (idle meaning, user is not leasing any buttons and counts and images are coming in continuously). Most important thing to notice here is the GUI freeze issue is not consistent here. There are several systems installed, and some places, the freeze (not responding) issue comes once in 2/3 weeks and in some places, once in 2 days. Waiting for the application to be responsive doesn’t help.

My main question is what is the main cause for the GUI to freeze and is there any checks to implement on serial thread and image capturing thread to avoid unnecessary data emissions.

vallabh
  • 140
  • 1
  • 10
  • Did you program a resampler on the capturing thread, so that the main thread is not flooded with incoming data ? Did you monitor the input rate ? – Arne J Jun 22 '19 at 06:14
  • @ArneJ I’m not sure I have done that, can you tell me how to check that? – vallabh Jun 22 '19 at 06:20
  • May be, this might help: [SO: How can I detect a hang in QEventLoop?](https://stackoverflow.com/a/25118309/7478597). As your freezes seem to occur sporadically, I would log times over a certain threshold. Are you sure that the freezes are not caused by deadlocks or some other kind of concurrency issues? (That's why I fear/hate multithreading - the errors occur sporadically and are hard to catch in the debugger.) – Scheff's Cat Jun 22 '19 at 06:29
  • 1
    @vallabh for starters, increment a single integer every time your capturing thread receives data. In another thread, run a loop that sleeps for one second and then prints the counter. Now do the same in the rendering (main) thread. If your application is stable the numbers should always be more or less the same, even after long sessions. – Arne J Jun 22 '19 at 06:44
  • The main should be with refreshing GUI very fast, which there is not enough time that GUI stays responsive. – Bahramdun Adil Jun 22 '19 at 06:52
  • @ArneJ I just had a similar idea: Use an `atomic` which is incremented in producer and decremented in GUI. It should be never negative (if initialized with 0) but reflects the number of pending images. – Scheff's Cat Jun 22 '19 at 07:40
  • @Scheff exactly :) to measure is to know ! – Arne J Jun 22 '19 at 07:48
  • 1
    _if the consumer thread doesn’t absorb the data fast enough then there can be a issue of GUI freezing._ In this case, the above idea could provide a proof as well as a fix. The producer (or even the consumer) might check how many pending images are not yet consumed and skip images in case. – Scheff's Cat Jun 22 '19 at 07:48
  • @Scheff what if I add a delay in consumer thread and emit the data every one second? – vallabh Jun 22 '19 at 07:49
  • _delay in consumer_? I think you mean delay in producer. If there are too many entries collecting in consumers event queue, the producer should be stalled... ;-) – Scheff's Cat Jun 22 '19 at 07:52
  • 1
    @vallabh bad idea, then you're still flooding the producer/capturing thread. Either control the input rate or resample (discard some frames). – Arne J Jun 22 '19 at 07:53
  • @Scheff what I meant is what if I capture the image every one second and emit the image to main thread, what will happen then? – vallabh Jun 22 '19 at 07:54
  • I assume this would lower the load in GUI thread. 1 s sounds long even for the not so fast Qt GUI. If you can live with the reduced frame rate, this might be an improvement. (At least, you might get an impression if the flooding of GUIs event loop was the actual reason of your issue...) – Scheff's Cat Jun 22 '19 at 07:56
  • @ArneJ I got it now, thanks – vallabh Jun 22 '19 at 07:57

1 Answers1

3

Sounds like you are experiencing a concurrency violation that doesn't necessarily result in crashing until things have been running long enough to finally hit the magic combination of events occurring at just the right time to bork things up.

You have three threads in your app: A GUI thread, a serial thread, and a camera thread. The serial and camera threads collect data from devices and then pass them on to the GUI thread for display. I presume the serial and camera threads don't share any data with each other, so there is no risk of problems there.

How are you passing data up from the serial and camera threads to the GUI thread? This is where you are probably having an issue.

Most complex data structures and Qt classes are not thread safe, meaning, they must never be read and written at the same time from two or more threads at the same time.

Here are some strategies for passing data between threads safely:

  1. An integer is atomic at the CPU's instruction set level, so you can safely read and write an integer (or any data type equal to or smaller than an integer, such as a bool, a single char, or a pointer) from multiple threads without having any inconsistent state occurring. You must declare such variables with C++'s std::atomic<> template to ensure that the compiler will not perform optimizations that break atomicity.

    Anything bigger/more complex than an integer runs the risk of one thread having written half of it's data to memory, while another thread concurrently reads out that half written data, resulting in very unexpected results, quite often crashing your application or getting something stuck in an infinite loop.

  2. Signals and slots in Qt are thread safe. One way to pass complex data between threads is to emit the data in one thread and have a slot in another thread receive that data. Qt takes care of any concurrency issues under the hood for you in this case. The only gotcha here is if the consumer thread(s) of the data can not absorb the data fast enough, Qt's event queue will get stuffed up with too much data and eventually your app will crash because new GUI events (such as mouse clicks, repaint events, etc) can no longer get through the clogged up event queue.

  3. You can use a QMutex to ensure that only one thread is reading or writing a complex data structure at one time. QMutex lets you block/halt execution in one or more threads while a single thread "holds" the mutex and allows it to execute, doing work on the data without any risk of other threads touching that data. When that one thread is done, it "releases" the mutex, which then allows one of the other threads to "hold" the mutex, resuming its execution so it can do work with the data.

As far as testing goes, typically the chances of your app crashing from a concurrency violation goes up the higher your data flow rates are. If you can artificially increase the rate of camera frames and "serial counts" being passed up to the GUI thread, you will be able to reproduce your application crash faster. Once you successfully solve your concurrency issues, you should be able to flood the system with data and never get it to crash.

K9spud
  • 323
  • 1
  • 7
  • Right now I’m using signal slot mechanism in the threads and as you said if the consumer thread doesn’t absorb the data fast enough then there can be a issue of GUI freezing. What is I add a delay in image capturing thread and emit the data every one second – vallabh Jun 22 '19 at 07:44
  • 2
    _An integer is atomic at the CPU's instruction set level, so you can safely read and write an integer_ That's a big misbelieve (I once had myself). You ignore that e.g. integers may be updated in CPU registers or cached perfectly. Hence, two threads might use two "incarnations" for the same variable. Furthermore, compiler might reorder assignments that might interfere with intended usage in separate threads. Therefore, `std` provides `std::atomic`. – Scheff's Cat Jun 22 '19 at 07:44
  • @vallabh, since you're working with so much data, if it were me, I'd stay away from using the signal/slot mechanism. Even if you slow down the camera thread pushing data onto the event queue to one frame per second, that's still a lot of data. The OS can and will halt your GUI thread for far longer than one second (during heavy disk I/O for example), leading to event queue clogging. What you really need is a design where the consumer can arbitrarily drop a number of frames when it gets too busy to fully handle the flow rate. – K9spud Jun 22 '19 at 08:30
  • @Scheff, good point! I added a note about std::atomic to my answer. Sometimes I'm thinking too much as an assembly language programmer on simpler architectures and forget about all the weird optimizations C/C++ compilers make plus all the fancy cache coherency stuff going on in modern multi-core CPUs. :) – K9spud Jun 22 '19 at 09:07
  • Too late to edit, but I should have said "where the PRODUCER can arbitrarily drop a number of frames when the consumer gets too busy to fully handle the flow rate." From my experience with flooding the event queue, once you've stuffed it, there doesn't seem be any recovering -- your app is crashed. That's why I don't like the idea of a fixed (albeit slower) producer push rate because it still has the potential to crash the app if the OS delays the consumer long enough. Dropping frames at the producer (skipping emits) when you detect the consumer is just starting to fall behind seems better. – K9spud Jun 22 '19 at 09:39