0

I have a java code to do a multiplication like this:

TickSize = minTick * AdjustFactor;

They are all double. When the code runs here, the miniSize value is 5.0E-7, AdjustFactor is 100. But I get a result as 4.9999999999999996E-5, instead of a 5.0E-5. This is very annoying and break the other module of my software.

The following is the debug message print out right after the above statement is executed.

minTick is 5.0E-7 Tick adjust factor is : 100.0 TickSize set to: 4.9999999999999996E-5

I wonder if anybody know what is going on? and what I can do to correct it to get result 5.0E-5?

Thanks,

Vincent W.
  • 1
  • 1
  • 1
    http://stackoverflow.com/questions/588004/is-floating-point-math-broken http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html http://floating-point-gui.de/ – Pshemo Jun 27 '16 at 19:49
  • You are working with double, so you can't ask a super precision. Bu, if you know this, you can change your other code to accept this error. – Benoît Verhaeghe Jun 27 '16 at 19:50
  • There are infinitely many real numbers, but there are fewer than 2^64 possible `double` values, so that means, there are an infinite number of reals that can not be exactly represented by a `double`. 5.0E-5 is one of them. You _can't_ set TickSize = 5.0E-5. But, you _can_ get it to print out as "5.0E-5" (or "5.00E-5", or "5.000E-5", or...) Read about the static function, `String.format(...)` and, the `java.util.Formatter` class that it uses. You'll want to use the 'e' conversion specifier, and you'll want to pay attention to the "precision" and "width" arguments. – Solomon Slow Jun 27 '16 at 20:18

0 Answers0