0

// why does this code gives this value? // shouldn't that the value be 0.9D ?

double test = 360.9D;

test -= 360D;

// test = 0.89999999999997726

Astro
  • 9
  • 1
  • 4

1 Answers1

1

This is due to inherent floating point imprecision that occurs when you want to store fractional base-10 numbers in binary. Wikipedia provides a good explanation

Basically you can't store all fractional base-10 numbers exactly in binary. The fractional portion is made up of sums of 2^n where n is a negative integer.

So to approximate 0.9, it will be 1/2 + 1/4 + 1/8 + 0/16 + 0/32 + 1/64 + ... and so on until the sum approximates 0.9, but you can see that the series will never exactly equal 0.9.

Luke
  • 743
  • 6
  • 20
  • 1
    https://floating-point-gui.de/ – Pete Feb 24 '23 at 20:32
  • 1
    https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html – Pete Feb 24 '23 at 20:33
  • @Pete were these comments meant for me or the question author? If they were meant for me, can you explain how I can enhance my answer? – Luke Feb 24 '23 at 20:34
  • 1
    The binary expansion of 0.9 decimal is 1/2 + 1/4 + 1/8 + 0/16 + 0/32 + 1/64 + ... No 1/16 or 1/32 in it. – Ben Voigt Feb 24 '23 at 20:40
  • 1
    Also, you can simplify "reciprocals of 2^n where n is a positive integer" to either "2^-n where n is a positive integer" or "2^n where n is a negative integer" – Ben Voigt Feb 24 '23 at 20:41
  • @Luke they were meant for the author, and also everyone who comes across this page. They enhance your answer by providing additional resources of which there are many available. – Pete Feb 24 '23 at 20:45
  • 2
    @Pete comments are generally to state requests to improve a post, if you have extra info/alternative answers provide one on your own (unless good answer already exists)... I.e. an example of proper comment would be "it is a bad idea to use word 'decimal' speaking about floating point numbers in C# because the `decimal` type in C#/.Net actually does support precise numbers and math with `decimal` would produce results expected by OP". – Alexei Levenkov Feb 24 '23 at 21:22
  • @Pete in that case these comments are probably better placed under the question rather than my answer – Luke Feb 24 '23 at 21:56
  • @AlexeiLevenkov Your example makes a fair criticism of the use of `decimal` in my answer. Replaced with base-10 – Luke Feb 24 '23 at 21:57
  • 1
    @Luke yes that would have been more cromulent, however, I made a mistake, so here we are. – Pete Feb 24 '23 at 21:59