1

Can anyone explain me why the result of the below does not equal zero?

? CSng("0.199881939681229")
? CSng(0.1998819)
? CSng(CSng(0.199881939681229) - CSng(0.1998819))

1st line returns 0.1998819 2nd line returns 0.1998819 too but the 3rd returns 4.470348E-08

Where is this figure coming from if both figures are represented exactly by the same value in a the single-precision data type.

Community
  • 1
  • 1
PaMcD
  • 105
  • 13
  • That is weird it seems it should be 3.968122E-08, Any reason you are using CSNG? when i use a variable type Single i get the correct answer. – Holmes IV Jul 03 '15 at 17:14
  • *note* i was able to duplicate this issue, weird, must be to do with how VBA is handling the conversions. when i inspect the variables they are all correct except after subtraction. – Holmes IV Jul 03 '15 at 17:22
  • see my comment below, both numbers are being appoximated by slightly different binary strings, even though VBA is returning the same value for both. – PaMcD Jul 06 '15 at 08:36

1 Answers1

0

seems that the other answer for this was deleted so i will respond myself.

VBA is returning 0.1998819 but it is not actually storing both numbers as the same binary.

see http://www.binaryconvert.com/result_float.html?decimal=048046049057057056056049057051057054056049050050057 and http://www.binaryconvert.com/result_float.html?decimal=048046049057057056056049057 .

Nearly identical but not.

mystery solved, VBA representation of floats in the immediate window was the issue.

PaMcD
  • 105
  • 13