Can somebody give me an in-depth explanation of what's going on? The system considers approximation right?(correct me if I'm wrong) I would like to know how the computer behaves in these kinds of situation. Thank you.
Asked
Active
Viewed 107 times
-1
-
https://floating-point-gui.de/ – Mat Nov 08 '19 at 17:30
-
[What Every Computer Scientist Should Know About Floating-Point Arithmetic](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html) – Jesper Juhl Nov 08 '19 at 17:31
1 Answers
1
Normal numbers in computers are stored with only so many bits of precision. A float in C++ is typically 4 bytes. 32 bits can't store that many 9s of precision, so the compiler does rounding to the precision it can handle.
Basically, you get approximately 10 digits of precision in total, and you have a lot more 9s than that.
Joseph Larson
- 8,530
- 1
- 19
- 36
-
2Please don't answer obvious dupes. We have a great canonical for Q's like this here: https://stackoverflow.com/questions/588004/is-floating-point-math-broken – NathanOliver Nov 08 '19 at 17:33