-1

I am getting a math error in vb.net.
Even tried in vb6 in my old project.
Even tried in NodeJs.(Asked a friend)
I am doing a simple stuff.

Msgbox(3.28 - 3.2)

This shows me 0.79999999999996(don't know how many 9 in between but this many for sure.)
Even tried Msgbox(Format(3.28 - 3.2,"0.00")) Still same answer.

In my Vb6 project, I get 7.9999999999996E-02 something.

In nodejs by my friend too get the same 0.7999…96

I need in vb.net so tried using Format but no proper answer. Maybe using format in wrong way.

What to do to get 0.08 ?

bonny
  • 688
  • 1
  • 14
  • 33
  • 1
    [Is floating point math broken?](https://stackoverflow.com/q/588004/7444103) -- [Why is floating point arithmetic imprecise?](https://stackoverflow.com/q/753948/7444103). – Jimi May 16 '19 at 17:49
  • `CDbl(3.28 - 3.2).Tostring("0.##", New System.Globalization.CultureInfo("en-US"))`... but there's also many other ways to do this. – Trevor May 16 '19 at 17:50
  • Put `Option Strict On` at the top of the file — forces you to fix your coding style. – LarsTech May 16 '19 at 17:50
  • @AndrewMorton No it's not, The question which you duplicated asked a reason for the error. I wanted a solution. – bonny May 17 '19 at 05:51
  • @bonny It's usually better to understand *why* something goes wrong so that a solution can be applied. The line `Msgbox(Format(3.28 - 3.2,"0.00"))` does show "0.08". Admittedly, the answer you need if you want to avoid errors from the approximation of base-10 fractions as binary was buried a bit at the end of one of the answers, disguised as a pizza cutter. – Andrew Morton May 17 '19 at 07:55
  • @AndrewMorton Okay – bonny May 17 '19 at 19:15

2 Answers2

2

If the numbers are being input by the user in base-10, and the result of the calculation is being displayed to the user in base-10, then you should store the values in Decimal variables rather than in Single or Double variables. Decimal stores numbers in a base-10 (decimal) floating point format whereas Single and Double store numbers in base-2 (binary) floating point format. So, if both the input and the output are base-10, a base-10 floating point variable will give you more expected results.

' Outputs "0.0799999999999996" because the literals are Double by default
Console.WriteLine(3.28 - 3.2) 

' Outputs "0.08" because the D suffix forces the compiler to interpret the literals as Decimal
Console.WriteLine(3.28D - 3.2D) ' Outputs

Floating points are always imprecise, since it's impossible to represent a value accurately in a digital format when it is a repeating decimal--at some point you have to stop the repetition and round it off. The problem is, which numbers infinitely repeat is different depending which base you are using, so by switching between bases, you can cause the value to be rounded in ways that you weren't necessarily expecting. So, for instance, even with Decimal, you can still get imprecise results, but at least they are rounded in the way that you expect them to be rounded, when doing base-10 math:

' Outputs "0.6666666666666666666666666667"
Console.WriteLine(2D / 3D)
Steven Doggart
  • 43,358
  • 8
  • 68
  • 105
  • Each time I read an explanation it gets a little clearer but this one really helped by pointing out the binary storage versus the base 10 storage. Thank you and upvote. – Mary May 17 '19 at 03:34
  • You are right but I am not going to change every Double to Decimal in my project. My project deals with numbers. Now if we move ahead and think that the error will come, a solution to again rectify the error would help and not by changing its data types. The Solution below by Greenfrogs, as you said he did rounded off the value and it worked. – bonny May 17 '19 at 06:02
-1

This is due to floating point arithmetic approximating the values. This can be solved by using System.Decimal.Round

Decimal.Round(0.575, 2, MidpointRounding.AwayFromZero)

Result: 0.58

Greenfrogs
  • 89
  • 1
  • 3