j=1
for i=1,10 do
j=j-0.1
print(j)
end
the output is:
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 1.3877787807814e-16
where the last entry should be 0 Note: 0.1-0.1 returns 0.0 but, repeatedly doing j=j-0.1 from j>=0.3 produces this result
Numbers in lua are doubles which is Double-precision floating-point format, floating-point numbers has floating precision error. In fact 0.1
is not exactly 0.1
, it's the closest representation of 0.1
.
Possible fix would be probably using integers or integer with separate decimal part. Related question with answer: C++ How to avoid floating-point arithmetic error
round off error!
0.1 cannot be represented exactly in IEEE 64 bit floating point 1/10 has a recurring representation in binary!