0

Consider the following:

console.log(0.1050.toPrecision(2));
console.log(0.1150.toPrecision(2));

// OUTPUT:
// "0.10"
// "0.12"

The rounding in javascripts 'toPrecision()' seems to me to be inconsistent. It's probably correct, but I don't understand. I expected, since JS is rounding halves up, that the first output would be "0.11", but it's not.

Can someone explain to me why this is? I can only reproduce this when a zero comes before the 5 to be rounded.

When using any 'significant digits calculator' around the web, these all come up with '0.11' as the correct answer for reducing to 2 significant digits.

kodesprog
  • 1
  • 1
  • 2
    `0.1050` is stored as `0.104999996721744537353515625` while `0.1150` is stored as `0.115000002086162567138671875`. See [here for the conversion](https://www.h-schmidt.net/FloatConverter/IEEE754.html). Related (but not a duplicate): [Is floating point math broken?](https://stackoverflow.com/q/588004) – VLAZ Feb 01 '23 at 12:00

0 Answers0