-1
j=1
for i=1,10 do 
    j=j-0.1
    print(j)
end

the output is:

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 1.3877787807814e-16

where the last entry should be 0 Note: 0.1-0.1 returns 0.0 but, repeatedly doing j=j-0.1 from j>=0.3 produces this result

oh_cakes
  • 1
  • 2

2 Answers2

1

Numbers in lua are doubles which is Double-precision floating-point format, floating-point numbers has floating precision error. In fact 0.1 is not exactly 0.1, it's the closest representation of 0.1.

Possible fix would be probably using integers or integer with separate decimal part. Related question with answer: C++ How to avoid floating-point arithmetic error

Spar
  • 1,582
  • 9
  • 16
0

round off error!

0.1 cannot be represented exactly in IEEE 64 bit floating point 1/10 has a recurring representation in binary!

Sadie Kaye
  • 27
  • 3