Stack Overflow
Stack Overflow

Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

Floating point numbers (typically meaning the IEEE standard) are inherently inexact and errors can compound, leading to edge cases in some decision processes, or numerical instability in certain algorithms.

Here is a mathematical treatment of the main problems.

1358 questions
3861
votes
33 answers

Is floating point math broken?

Consider the following code: 0.1 + 0.2 == 0.3 -> false 0.1 + 0.2 -> 0.30000000000000004 Why do these inaccuracies happen?
Cato Johnston
  • 44,131
  • 10
  • 39
  • 42
420
votes
12 answers

How dangerous is it to compare floating point values?

I know UIKit uses CGFloat because of the resolution independent coordinate system. But every time I want to check if for example frame.origin.x is 0 it makes me feel sick: if (theView.frame.origin.x == 0) { // do important operation } Isn't…
Proud Member
  • 40,078
  • 47
  • 146
  • 231
306
votes
6 answers

Why are these numbers not equal?

The following code is obviously wrong. What's the problem? i <- 0.1 i <- i + 0.05 i ## [1] 0.15 if(i==0.15) cat("i equals 0.15") else cat("i does not equal 0.15") ## i does not equal 0.15
dplanet
  • 5,273
  • 9
  • 29
  • 44
193
votes
21 answers

What's wrong with using == to compare floats in Java?

According to this java.sun page == is the equality comparison operator for floating point numbers in Java. However, when I type this code: if(sectionID == currentSectionID) into my editor and run static analysis, I get: "JAVA0078 Floating point…
user128807
  • 10,447
  • 17
  • 53
  • 72
164
votes
4 answers

Why does the floating-point value of 4*0.1 look nice in Python 3 but 3*0.1 doesn't?

I know that most decimals don't have an exact floating point representation (Is floating point math broken?). But I don't see why 4*0.1 is printed nicely as 0.4, but 3*0.1 isn't, when both values actually have ugly decimal representations: >>>…
Aivar
  • 6,814
  • 5
  • 46
  • 78
133
votes
5 answers

Dealing with float precision in Javascript

I have a large amount of numeric values y in javascript. I want to group them by rounding them down to the nearest multiple of x and convert the result to a string. How do I get around the annoying floating point precision? For example: 0.2 +…
Jeroen Ooms
  • 31,998
  • 35
  • 134
  • 207
132
votes
12 answers

Is it possible to get 0 by subtracting two unequal floating point numbers?

Is it possible to get division by 0 (or infinity) in the following example? public double calculation(double a, double b) { if (a == b) { return 0; } else { return 2 / (a - b); } } In normal cases it…
Thirler
  • 20,239
  • 14
  • 63
  • 92
129
votes
32 answers

Truncate (not round off) decimal numbers in javascript

I am trying to truncate decimal numbers to decimal places. Something like this: 5.467 -> 5.46 985.943 -> 985.94 toFixed(2) does just about the right thing but it rounds off the value. I don't need the value rounded off. Hope this is possible…
kcssm
  • 1,727
  • 3
  • 16
  • 19
98
votes
4 answers

What's the closest double to 1.0, that isn't 1.0?

Is there a way to programmatically get the double that is closest to 1.0, but isn't actually 1.0? One hacky way to do this would be to memcpy the double to a same-sized integer, and then subtract one. The way IEEE754 floating-point formats work,…
jorgbrown
  • 1,993
  • 16
  • 23
96
votes
8 answers

PHP - Floating Number Precision

$a = '35'; $b = '-34.99'; echo ($a + $b); Results in 0.009999999999998 What is up with that? I wondered why my program kept reporting odd results. Why doesn't PHP return the expected 0.01?
dcmoody
  • 1,295
  • 1
  • 11
  • 15
83
votes
6 answers

Large numbers erroneously rounded in JavaScript

See this code: var jsonString = '{"id":714341252076979033,"type":"FUZZY"}'; var jsonParsed = JSON.parse(jsonString); console.log(jsonString, jsonParsed); When I see my console in Firefox 3.5, the value of jsonParsed is the number rounded: Object…
Jaanus
  • 17,688
  • 15
  • 65
  • 110
75
votes
3 answers

Why 0.1 + 0.2 == 0.3 in D?

assert(0.1 + 0.2 != 0.3); // shall be true is my favorite check that a language uses native floating point arithmetic. C++ #include int main() { printf("%d\n", (0.1 + 0.2 != 0.3)); return…
Stas
  • 11,571
  • 9
  • 40
  • 58
70
votes
6 answers

(.1f+.2f==.3f) != (.1f+.2f).Equals(.3f) Why?

My question is not about floating precision. It is about why Equals() is different from ==. I understand why .1f + .2f == .3f is false (while .1m + .2m == .3m is true). I get that == is reference and .Equals() is value comparison. (Edit: I know…
LZW
  • 1,035
  • 2
  • 10
  • 13
65
votes
5 answers

Is floating point arbitrary precision available?

Just for fun and because it was really easy, I've written a short program to generate Grafting numbers, but because of floating point precision issues it's not finding some of the larger examples. def isGrafting(a): for i in xrange(1,…
OmnipotentEntity
  • 16,531
  • 6
  • 62
  • 96
63
votes
7 answers

negative zero in python

I encountered negative zero in output from python; it's created for example as follows: k = 0.0 print(-k) The output will be -0.0. However, when I compare the -k to 0.0 for equality, it yields True. Is there any difference between 0.0 and -0.0 (I…
max
  • 49,282
  • 56
  • 208
  • 355
1
2 3
…
90 91