0
math.floor(10**62 * (1-3/(10**31)))
output: 100000000000000003502199685943161173046080317798311825604870144

math.floor(10**62-3*(10**31))
output:99999999999999999999999999999970000000000000000000000000000000


10**62 * (1-3/(10**31)) - (10**62-3*(10**31))
output: 0.0

I wanted to check my answer for a number theory problem and I'm getting two different floors of the same number. What is going wrong?

Mach
  • 1
  • ``float`` is limited precision. Their difference being zero does not mean equality, it means the difference is smaller than the precision. Compare ``(10**62 * (1-3/(10**31))) == (10**62-3*(10**31))`` and ``int(10**62 * (1-3/(10**31))) - int(10**62-3*(10**31))``. – MisterMiyagi Dec 15 '20 at 10:09
  • Does this answer your question? [Is floating point math broken?](https://stackoverflow.com/questions/588004/is-floating-point-math-broken) – MisterMiyagi Dec 15 '20 at 10:09

0 Answers0