0

I tried this:

t = (Math.log10(7)+3)/2
10.0**(2.0*t-3.0)
=> 7.000000000000002

The result should be 7 not 7.000000000000002. Why?

rtacconi
  • 14,317
  • 20
  • 66
  • 84
  • Possible duplicate of [Is floating point math broken?](http://stackoverflow.com/q/588004/1157054). – Ajedi32 Aug 11 '14 at 19:47
  • It's not Ruby specific. It's a very common issue with floating point arithmetic. I found a page in the [Python docs](https://docs.python.org/2/tutorial/floatingpoint.html) which covers it with some examples, but you can find plenty of other sources (also on this site). – Daniël Knippers Aug 11 '14 at 19:48
  • See also: [Issue with precision of Ruby math operations](http://stackoverflow.com/questions/3777119/issue-with-precision-of-ruby-math-operations) – Ajedi32 Aug 11 '14 at 19:52

0 Answers0