Why is this basic number comparison producing an illogical result?

Question

While hunting a weird bug in my VB.NET application, I tracked it down to a shockingly puzzling detail. Here is a simple test code:

If 0.01 > 0.12 - 0.11 Then Debug.Print("what the hell")

0.12-0.11 is 0.01... Which is equal to the left side of the comparison. However, when I run this, the debug prints "what the hell"... Because seriously, what the hell. These numbers are equal.

Additionally, if I have a cycle like this:

Dim count As Integer = 0
For i As Double = 0.11 to 0.12 Step 0.01
   count += 1
Next
Debug.Print(count)

It prints 1, meaning the cycle is executed only once, while it should execute twice.

Surprisingly, if I change 0.11, 0.12 and 0.01 in the above examples to 0.1, 0.2 and 0.1, then first example doesn't print anything, and the second example prints 2, as it should.

What is going on here? Am I missing something incredibly obvious, or is this some kind of floating point error or something?

`If CSng(0.01) > CSng(0.12 - 0.11) Then (...)`. Declare the number as Double instead and look at the Locals Window while debugging, so you can see the actual values. Your `0.12 - 0.11` is `0.009999999999999995`. [Is floating point math broken?](https://stackoverflow.com/q/588004/7444103). [Difference between decimal, float and double in .NET?](https://stackoverflow.com/a/618596/7444103). — Jimi, Feb 07 '19 at 23:38
Thank you, but what can I do to avoid having to CSng everything in my code when I need to make comparisons like this? I tried declaring these as doubles... They show as 0.01 > 0.01 in debug, and yet this comparison returns True. — Justinas Rubinovas, Feb 07 '19 at 23:49
Assign the values to a local variable, so you can inspect the value(s) when an operation is performed. Don't use floating point numbers directly; they're usually interpreted as Double, but you can't trust an automatic inference. **You** should determine what Types you're using. Use the informations provided in the linked questions to decide what is the Type you need. If you haven't, set `Option Strinct ON` for every project using the Visual Studio options tool. — Jimi, Feb 07 '19 at 23:58
Okay, thank you. But what if I need to use floating point numbers in a cycle? How do I do a cast there (like in the second example I posted) to avoid these floating point errors? Sometimes using floating point numbers in cycles are unavoidable in my code. — Justinas Rubinovas, Feb 08 '19 at 00:00
Assign the values to a local variable. Or set the value Type explicitly: `0.01D` = Decimal, `0.01#` = Double, `0.01F` = Float (Single). If you use local variables, then, while debugging, you can use the Locals Window to have a better view of the values that are being actually processed. — Jimi, Feb 08 '19 at 00:04
BTW, see the difference between these assignments: `Dim number1 As Single = 0.12F - 0.11F`, `Dim number2 As Single = 0.12 - 0.11F`, `Dim number3 As Single = 0.12F - 0.11`, `Dim number4 As Single = CSng(0.12 - 0.11)`. If you have read what's in the links I posted, you won't be surprised. — Jimi, Feb 08 '19 at 00:31
Possible duplicate of [Is floating point math broken?](https://stackoverflow.com/questions/588004/is-floating-point-math-broken) — Blackwood, Feb 08 '19 at 00:33
Jimi, the debug gives me headaches. I declared it like this: `Dim valA As Double = 0.01, Dim valB As Double = 0.12, Dim valC As Double = 0.11 Dim Difference As Single = valB - valC` , and yet after this `valA > Difference` returns True. When I check these local variables in Locals Window, both valA and Difference are exactly 0.01. Am I still missing something? — Justinas Rubinovas, Feb 08 '19 at 01:32
I told you to set `Option Strict On`, didn't I? With it `On`, you can't assign this: `Dim Difference As Single = valB - valC`, because you're assigning a Double Type value to a Single Type variable. Try this: `Dim valA As Single = 0.01 Dim valB As Double = 0.12 Dim valC As Double = 0.11 Dim Difference As Single = CSng(valB - valC) Dim result As Boolean = valA > Difference`. This is functionally equivalent, if you have all Single or Double values, to `Dim Difference As Single = CSng(Math.Round((valB - valC), 2))` — Jimi, Feb 08 '19 at 02:31
If you use all Double values, setting `Dim Difference As Double = Math.Round((valB - valC), 2)`, inspect `valA`, `valB` and `valC` **after** they have been assigned a value. With `Option Strict ON`, I mean. See what is the actual value those variables are carrying. — Jimi, Feb 08 '19 at 02:45

score 1 · Answer 1 · answered Feb 08 '19 at 00:04

1

You're getting these problems because you're using floating-point types, which use base 2, and base 2 can't represent some fractional values exactly.

That's why fixed-point types like Decimal were devised. If your example code is reworked for fixed-point (using Decimal), it gives the expected results.

    ' Specify Decimal constants and this will worked as anticipated.
    If 0.01D > 0.12D - 0.11D Then Debug.Print("what the hell")

    ' 0.12-0.11 Is 0.01... Which Is equal to the left side of the comparison.
    ' However, when I run this, the debug prints "what the hell"... Because 
    ' seriously, what the hell. These numbers are equal.

    ' Additionally, If I have a cycle Like this

    Dim count As Integer = 0
    For i As Decimal = 0.11D To 0.12D Step 0.01D
        count += 1
    Next

    Debug.Print(count) ' Prints 2

answered Feb 08 '19 at 00:04

R.J. Dunnill

2,049
3
10
21

Thank you, but using decimals is incredibly slow, compared to floating-point types... And performance is a serious consideration in my application. What can I do to avoid these floating point errors when, for example, using a Double to iterate in the For cycle? – Justinas Rubinovas Feb 08 '19 at 00:16
Rounding issues are a drawback of floating-point types. AFAIK numeric constants with decimal points (like 0.01) are doubles and will exhibit the problem behavior. – R.J. Dunnill Feb 08 '19 at 00:26
@JustinasRubinovas: If you're working with money you can consider using integers for your calculations (number of cents instead of number of dollars). Obviously this doesn't work if you can have fractions of cents. – Meta-Knight Feb 08 '19 at 17:14

score 0 · Answer 2 · answered Feb 08 '19 at 02:54

How about Integer arithmetic?

Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
    Dim count As Integer = 0
    For i As Integer = CInt(0.11 * 100) To CInt(0.12 * 100) Step CInt(0.01 * 100)
        count += 1
    Next
    Debug.Print(count.ToString)
End Sub

Private Sub Button2_Click(sender As Object, e As EventArgs) Handles Button2.Click
    If CInt(0.01 * 100) > CInt(0.12 * 100) - CInt(0.11 * 100) Then
        Debug.Print("what the hell")
    Else
        Debug.Print("It's Ok")
    End If
End Sub

score 0 · Answer 3 · answered Feb 08 '19 at 15:28

If you can't use Decimal for your calculations, then you must write your code to account for the fact that binary floating point types can't represent some fractional values exactly. So rather than checking if the numbers are equal, you check if they are nearly equals.

You can use code like the following (taken from this article by Michael Borgwardt:

This is the VB translation, but not tested extensively.

Public Shared Function NearlyEqual(a As Double, b As Double, epsilon As Double) As Boolean
    Dim absA As Double = Math.Abs(a)
    Dim absB As Double = Math.Abs(b)
    Dim diff As Double = Math.Abs(a - b)

    If (a = b) Then
        'shortcut, handles infinities
        Return True
    Else 
        If (a = 0 OrElse b = 0 OrElse diff < Double.Epsilon) 
            'a or b is zero or both are extremely close to it
            'relative error is less meaningful here
            Return diff < epsilon
        Else
            'use relative error
            Return diff / (absA + absB) < epsilon
        End If
    End If
End Function

Why is this basic number comparison producing an illogical result?

3 Answers3