0

Shouldn't the floating point error be present in both calculations? Why does the error get dropped in 5.1-0.1?

enter image description here

Sean
  • 213
  • 1
  • 3
  • 11
  • 1
    There probably is an error, but it may occur after around 15 digits so is not accounted for. If the error occurs at a larger value like `1e-10` then it is counted. That’s the ‘simple’ explanation. – N Chauhan Oct 30 '18 at 16:38
  • 2
    Actually, the error in `5.1` occurs at the 16th decimal place and the error in `0.1` occurs at the 18th decimal place. Subtracting those two would give an error in the 16th decimal place, but that gets rounded off to only 8 bytes so the final answer in the 8-byte floating point register is indeed exactly `5.0` with no error. (At least, that is what happens with my 64-bit CPython installation on my Intel CPU and Windows 10 system.) This can be seen by playing with the `Decimal` type in the `decimal` module. – Rory Daulton Oct 30 '18 at 17:53

0 Answers0