I run a function in SQL Server that returns some calculated values. When testing the results in FitNesse using dbFit it claims that the values are not the same (see the screenshot below).
The values are returned as floats and fails on the comparison of all decimals - even though it only displays two. Is there a way to limit the range of decimals that gets compared?
You should NEVER compare floats, at least not strictly. Thats because of the binary system imperfections.
float a = 0.15 + 0.15
float b = 0.1 + 0.2
if(a == b) // can be false!
if(a >= b) // can also be false!
Since the significant digits in a real number are limited, we cant store real numbers perfectly. Theres almost always an error. Try to write 0.1 in binary! What we see printed on the screen as 0.1 is really only an approximation. Its a vast subject, you might want to get into numerical analysys for details.
On a more general note, you'll find some info here: http://www.cygnus-software.com/papers/comparingfloats/comparingfloats.htm
as for your problem:
You should use ~= (approx. equal). You could find more info here: http://fitnesse.org/FitNesse.UserGuide.SliM.ValueComparisons
