Anything related to the precision of a floating-point number representation. The term precision refers to the number of significant digits a representation can hold. This is NOT the same as the "accuracy", which concerns errors in performing calculations, although it may be sometimes related.
Questions tagged [floating-point-precision]
551 questions
156
votes
9 answers
Printf width specifier to maintain precision of floating-point value
Is there a printf width specifier which can be applied to a floating point specifier that would automatically format the output to the necessary number of significant digits such that when scanning the string back in, the original floating point…
Vilhelm Gray
- 11,516
- 10
- 61
- 114
86
votes
4 answers
What's the C++ suffix for long double literals?
In C++ (and C), a floating point literal without suffix defaults to double, while the suffix f implies a float. But what is the suffix to get a long double?
Without knowing, I would define, say,
const long double x =…
Walter
- 44,150
- 20
- 113
- 196
66
votes
4 answers
Python: Find index of minimum item in list of floats
How can I find the index of the minimum item in a Python list of floats? If they were integers, I would simply do:
minIndex = myList.index(min(myList))
However, with a list of floats I get the following error, I assume because float equality…
thornate
- 4,902
- 9
- 39
- 43
58
votes
6 answers
Set specific precision of a BigDecimal
I have an XSD that requires me to use a BigDecimal for a lat/lon. I currently have the lat/lon as doubles, and convert them to BigDecimal, but I am only required to use about 12 places of precision. I have not been able to figure out how to set…
Jason
- 2,147
- 6
- 32
- 40
50
votes
1 answer
Does parseDouble exist in JavaScript?
In JavaScript, I have a number which is 21 digits, and I want to parse it.
Does a parseDouble method exist in JavaScript?
Priyesh Shah
- 609
- 2
- 6
- 8
46
votes
4 answers
Can I specify a numpy dtype when generating random values?
I'm creating a numpy array of random values and adding them to an existing array containing 32-bit floats. I'd like to generate the random values using the same dtype as the target array, so that I don't have to convert the dtypes manually.…
lmjohns3
- 7,422
- 5
- 36
- 56
43
votes
2 answers
How to avoid floating point errors?
I was trying to write a function to approximate square roots (I know there's the math module...I want to do it myself), and I was getting screwed over by the floating point arithmetic. How can you avoid that?
def sqrt(num):
root = 0.0
while…
temporary_user_name
- 35,956
- 47
- 141
- 220
42
votes
4 answers
pow() seems to be out by one here
What's going on here:
#include
#include
int main(void) {
printf("17^12 = %lf\n", pow(17, 12));
printf("17^13 = %lf\n", pow(17, 13));
printf("17^14 = %lf\n", pow(17, 14));
}
I get this output:
17^12 =…
jsj
- 9,019
- 17
- 58
- 103
39
votes
10 answers
Is floating point precision mutable or invariant?
I keep getting mixed answers of whether floating point numbers (i.e. float, double, or long double) have one and only one value of precision, or have a precision value which can vary.
One topic called float vs. double precision seems to imply that…
Wandering Fool
- 2,170
- 3
- 18
- 48
32
votes
3 answers
Correct use of std::cout.precision() - not printing trailing zeros
I see many questions about the precision number for floating point numbers but specifically I want to know why this code
#include
#include
int main()
{
int a = 5;
int b = 10;
std::cout.precision(4);
std::cout <<…
mahmood
- 23,197
- 49
- 147
- 242
29
votes
3 answers
Get next smallest Double number
As part of a unit test, I need to test some boundary conditions. One method accepts a System.Double argument.
Is there a way to get the next-smallest double value? (i.e. decrement the mantissa by 1 unit-value)?
I considered using Double.Epsilon but…
Dai
- 141,631
- 28
- 261
- 374
28
votes
1 answer
Why does 0.06 + 0.01 = 0.07 in ColdFusion?
Why don't math operations in ColdFusion seem to be affected by floating point math issues? Take the code:
result = 0.06 + 0.01;
writedump(result);
writedump(result.getClass().getName());
Which outputs
0.07
java.lang.Double
However the…
bittersweetryan
- 3,383
- 5
- 28
- 42
27
votes
5 answers
Why does for loop using a double fail to terminate
I'm looking through old exam questions (currently first year of uni.) and I'm wondering if someone could explain a bit more thoroughly why the following for loop does not end when it is supposed to. Why does this happen? I understand that it skips…
Patidati
- 1,048
- 2
- 12
- 19
24
votes
3 answers
Is the most significant decimal digits precision that can be converted to binary and back to decimal without loss of significance 6 or 7.225?
I've come across two different precision formulas for floating-point numbers.
⌊(N-1) log10(2)⌋ = 6 decimal digits (Single-precision)
and
N log10(2) ≈ 7.225 decimal digits (Single-precision)
Where N = 24 Significant bits (Single-precision)
The…
Wandering Fool
- 2,170
- 3
- 18
- 48
24
votes
3 answers
How to safely floor or ceil a CGFloat to int?
I often need to floor or ceil a CGFloat to an int, for calculation of an array index.
The problem I permanently see with floorf(theCGFloat) or ceilf(theCGFloat) is that there can be troubles with floating point inaccuracies.
So what if my CGFloat is…
Proud Member
- 40,078
- 47
- 146
- 231