Unable to compare calculated number with idential number?

Question

I'm working on a vector class. While testing the vector class, I ran into a failed test case when comparing the magnitude of a unit vector with 1.

When comparing them, they both appear to be 1, so what would cause this test to fail?

I have cut out as much as possible to get down to the root cause of the problem. I could just cast the numbers to strings and compare those, but that would only fix the test case, allowing the problem to show up again later down the line. I'm using Lua 5.1 interpreter (to identify the root cause), bundled with LuaRocks.

local v = 0.70710678118655^2 + 0.70710678118655^2
print(v, v == 1)

v == 1 should be true, not false.

Related: [Is floating point math broken?](https://stackoverflow.com/questions/588004/is-floating-point-math-broken) — aschepler, Jun 20 '19 at 03:24
I did not think it was a floating point error since it still displayed both numbers as `1`. Would it just not display the remaining digits if they are small enough? — cj-, Jun 20 '19 at 03:39

score 1 · Accepted Answer · answered Jun 20 '19 at 21:25

When converting a float to a string, Lua rounds off to only 15 significant digits. Try making it 17 digits to get an exact representation.

local v = 0.70710678118655^2 + 0.70710678118655^2
print(("%.17g"):format(v), v == 1)

Output:

1.0000000000000071  false

score 0 · Answer 2 · answered Jun 20 '19 at 11:40

Of course this is an examble of Is floating point math broken.

For more explanation: execute this code

local half = 0.50
local doubleHalf = half + half
if doubleHalf == 1 then
    print('doubleHalf is one')
else
    print('doubleHalf differs '.. doubleHalf - 1 ..'from one')
end
--> doubleHalf is one

local rootHalf = math.sqrt(half)
print('rootHalf is a '.. type(rootHalf) ..' with value', rootHalf)
--> rootHalf is a number with value    0.70710678118654

local square = rootHalf^2 
print('square is a '.. type(square) ..' with value', square)
--> square is a number with value    0.5

if square == half then
    print('square is half')
else
    print('square differs '.. square - half ..' from half')
end
--> square differs 1.110223E-16 from half

vector_length = square + square
if vector_length == 1 then
    print('vector length is one')
else
    print('vector length differs '.. vector_length - 1 ..' from one')
end
--> vector length differs 2.220446E-16 from one

Any computer working with values will calculate with a limited precission.

Unable to compare calculated number with idential number?

2 Answers2