How to Find Code Resulting in Inconsistent Results

Question

I have a C++ program that processes images and tracks objects in them, using OpenCV. For the most part, it works well; however the results that I get are inconsistent. That is, approximately 10% of the time, I am getting slightly different output values and I cannot figure out why. I do not have any calls to random; I have run valgrind to look for uninitialized memory; I have run clang-tools static analysis on it. No luck. The inconsistent runs have one of several different outputs, so they are not completely random.

Is there a tool that will show me where two runs diverge? If I run gprof or maybe cflow, can I compare them and see what was different? Is there some other tool or process I can use?

Edit: Thank you for the feedback. I believe that it is due to threading and a race condition; the suggestion was very helpful. I am currently using advice from: Ways to Find a Race Condition

some CV processing isn't always consistent, as long as the result is valid, there is no problem. — apple apple, Oct 21 '19 at 14:47
Are you using any multithreading? Any floating point math? I assume you are using the exact same inputs each time (on the same machine/compiler/build)? — Max Langhof, Oct 21 '19 at 14:47

score 0 · Answer 1 · answered Apr 15 '20 at 12:00

Answering my own question, so that someone can see it. It was not a race condition:

The underlying problem was that we were using HOG descriptors with the incorrect parameters.

As it says in the documentation (https://docs.opencv.org/3.4.7/d5/d33/structcv_1_1HOGDescriptor.html), when you call cv::HOGDescriptor::compute, and you pass in a win_stride, it has to be a multiple of block stride. Similarly, block stride must be a multiple of cell stride. We did not set these properly. The end result is that sometimes (about 10% of the time), memory was being overwritten or otherwise corrupted. It didn't throw an error, and the results were almost correct all the time, but they were subtly different.

How to Find Code Resulting in Inconsistent Results

1 Answers1