0

I am building a Metrics Converter site where precise calculations matter a lot and I noticed a strange issue when performing these metrics calculations.

Let me post a few instances which I have come across and any solution would be appreciated.

I have a Centimeter to Meter Conversion Page where I take the user input and divide by 100 to get the meter output. It works properly except for a few strange numbers.

Eg - 0.009 cm would be 0.00009 Meters

The formula used would be 0.009/100 = 0.00009 Meters

However, JavaScript returns a weird result

  1. 0.009/100 returns as 0.00008999999999999999
  2. 0.007/100 returns as 0.00007000000000000001
  3. 0.00009/100 returns as 9.000000000000001e-7

The same can be verified in the console or on the website cm to meters also by providing the centimeter value.

Now the site requires precise calculations for these metrics/units it would be a bad experience to use if it displays a strange result with long numbers. I am sure that it will happen with many other floating point numbers and other conversion formulas that is been used.

I have noticed the same is not happening in Python, C# language as it returns accurately.

Any solutions or workaround is appreciated to solve this issue

Srinivas Ramakrishna
  • 1,294
  • 13
  • 21
  • 1
    Actually, the same happens with Python `print(0.009/100 == 0.00008999999999999999) # True` `print(0.007/100 == 0.00007000000000000001) # True` `print(0.00009/100 == 9.000000000000001e-7) # True` `print(0.009/100 == 0.00009) # False` `print(0.007/100 == 0.00007) # False` `print(0.00009/100 == 9e-7) # False` – qrsngky Jul 18 '22 at 08:49
  • I believe i need to use a parseFloat function with `toFixed()` which solves the issue. `parseFloat((0.009/100).toFixed(10))` – Srinivas Ramakrishna Jul 18 '22 at 09:05

0 Answers0