0

A very weird issue happens when substracting two numbers in vb.net It returns a wrongful outcome for a very simple math equation : I am using visual studio 2019 I don't want to use any math.round to fix the value for such simple equation Any ideas why that issue happens?

Dim diff As Decimal = 100.1 - 100
MsgBox(diff)

it returns 0.0999999999999943

MsgBox(diff)

Ross Presser
  • 6,027
  • 1
  • 34
  • 66
Mouhammad Abbasi
  • 1
  • 2
  • 2
    Duplicate of https://stackoverflow.com/questions/588004/is-floating-point-math-broken – esqew Jan 31 '22 at 19:55
  • Not a duplicate of that question, but that question's accepted answer generally explains what the issue is. But it doesn't really provide a solution. – djv Jan 31 '22 at 20:19
  • Before you continue on with anymore software development, you should read [this article](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html), which will explain everything you ever need to know about floating point numbers and computers. – Icemanind Jan 31 '22 at 21:02

1 Answers1

2

There is a lot more going on in your math than you assume! You have three different primitive types there:

  • 100.1 is a Double
  • 100 is an Integer
  • diff is a Decimal

If you put Option Strict On at the top of your code, your code will not compile. This will tell you

Option Strict On disallows implicit conversions from 'Double' to 'Decimal'.

So you must either explicitly cast (this is what your Option Strict Off version is doing implicitly)

Dim diff As Decimal = CDec(100.1 - 100)

but since it's the same thing you are already doing implicitly, you will get that same floating point issue. The better thing to do is to start with a Decimal instead of Double in the first place, such as

Dim diff = 100.1D - 100

By using the Decimal literal D at the end of the floating point number it actually does Decimal math, and the result:

enter image description here

djv
  • 15,168
  • 7
  • 48
  • 72
  • thanks djv , works like magic here is my final code : DBG1.Rows(row).Cells(8).Value = CDec(Val(DBG1.Rows(row).Cells(6).Value)) - (CDec(Val(DBG1.Rows(row).Cells(5).Value)) * CDec(Val(DBG1.Rows(row).Cells(4).Value))) – Mouhammad Abbasi Jan 31 '22 at 20:29
  • @MouhammadAbbasi that doesn't look right. You've got to stop using outdated functions such as `MsgBox` and `Val`, instead use `MessageBox.Show` and the appropriate numeric parsing / casting functions. Based on your comment, you may still have an issue with floating point conversions. Your question should include that code. – djv Jan 31 '22 at 20:33
  • noted , thanks alot – Mouhammad Abbasi Jan 31 '22 at 20:44
  • @MouhammadAbbasi Use this instead: `Decimal.Parse(DBG1.Rows(row).Cells(6).Value)` to parse the string directly into decimal, then you won't lose precision when converting first to Double with `Val` then Decimal with `CDec` – djv Jan 31 '22 at 20:49