2

I'm having an hard time to understand what's going on behind the scene while performing VB6 double-precision number calculations.

Screenshot 1 : As you can see, I have a sum of 2 variables ("zpoint" and "zvecteur", visually shown by the IDE as 18881.132087 and -19091.737188)

  • When summed using the variables, the result is -210.605101000001 (d)
  • When "manually" summed outside of the variables, the result is -210.605100999997 (d2) 1st screenshot

I don't understand why there's a difference, because as you can see on the 2nd screenshot, for a quite similar operation, the result is strictly the same 2nd screenshot

I'm aware that there's probably a difference between what's visually shown by Visual Basic and what's really stored in memory but it's a problem because I'll later perform iterative operations on those values, and final results can really be different because of that. I have to mention that difference in precision only happens on a few values. I did not found any pattern to understand what's going on.

Any help would be much appreciated :) Thank you very much !

Guizmox
  • 34
  • 4
  • 7
    Quick answer - what you're seeing is normal. First, Floating point variables (Double, Single) are _approximate_, not exact. See https://stackoverflow.com/questions/7372670. Second, the math is performed with internal binary representations of the floating point values, but you're displaying the results in decimal representation, that conversion is itself not exact. Third, your "manual" summing starts with decimal, is converted to binary, then sum is converted back to decimal, thus it has 'extra' conversions which can introduce inexactness at each stage. – MarkL Aug 23 '22 at 16:31
  • I have to mention that if I launch the operation from a Winform app (by copying the entire Calculs.cs class), the result is -210.605101000001, but if I launch it from a Console app, the result is -210.605100999997 (same computer, same IDE). Do you think there's something that can be linked with your explanation in some way ? – Guizmox Aug 23 '22 at 16:42
  • Also found this article : https://stackoverflow.com/questions/1421520/formatting-doubles-for-output-in-c-sharp/1658420#1658420 – Guizmox Aug 24 '22 at 07:48
  • Please post the actual code and not screenshots – StayOnTarget Aug 24 '22 at 11:41

0 Answers0