Stack Overflow
Stack Overflow

Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

In general, the precision of an approximate real number is the effective number of decimal digits in it which are treated as significant for computations. The accuracy is the effective number of these digits which appear to the right of the decimal point.

For precision in Information Retrieval (the ratio of relevant items to total items retrieved) please use the tag .

4478 questions
3861
votes
33 answers

Is floating point math broken?

Consider the following code: 0.1 + 0.2 == 0.3 -> false 0.1 + 0.2 -> 0.30000000000000004 Why do these inaccuracies happen?
Cato Johnston
  • 44,131
  • 10
  • 39
  • 42
2314
votes
36 answers

Limiting floats to two decimal points

I want a to be rounded to 13.95. I tried using round, but I get: >>> a 13.949999999999999 >>> round(a, 2) 13.949999999999999 For the analogous issue with the standard library Decimal class, see How can I format a decimal to always show 2 decimal…
kevin
997
votes
7 answers

decimal vs double! - Which one should I use and when?

I keep seeing people using doubles in C#. I know I read somewhere that doubles sometimes lose precision. My question is when should a use a double and when should I use a decimal type? Which type is suitable for money computations? (ie. greater…
Soni Ali
  • 18,464
  • 16
  • 44
  • 53
526
votes
14 answers

What is the difference between float and double?

I've read about the difference between double precision and single precision. However, in most cases, float and double seem to be interchangeable, i.e. using one or the other does not seem to affect the results. Is this really the case? When are…
VaioIsBorn
  • 7,683
  • 9
  • 31
  • 28
485
votes
14 answers

JavaScript displaying a float to 2 decimal places

I wanted to display a number to 2 decimal places. I thought I could use toPrecision(2) in JavaScript . However, if the number is 0.05, I get 0.0500. I'd rather it stay the same. See it on JSbin. What is the best way to do this? I can think of coding…
alex
  • 479,566
  • 201
  • 878
  • 984
442
votes
16 answers

How do I print a double value with full precision using cout?

In my earlier question I was printing a double using cout that got rounded when I wasn't expecting it. How can I make cout print a double using full precision?
Restore the Data Dumps
  • 38,967
  • 12
  • 96
  • 122
300
votes
4 answers

How do I interpret precision and scale of a number in a database?

I have the following column specified in a database: decimal(5,2) How does one interpret this? According to the properties on the column as viewed in SQL Server Management studio I can see that it means: decimal(Numeric precision, Numeric…
mezoid
  • 28,090
  • 37
  • 107
  • 148
286
votes
28 answers

How do you round a double in Dart to a given degree of precision AFTER the decimal point?

Given a double, I want to round it to a given number of points of precision after the decimal point, similar to PHP's round() function. The closest thing I can find in the Dart docs is double.toStringAsPrecision(), but this is not quite what I need…
Lucas Meadows
  • 2,939
  • 2
  • 14
  • 4
284
votes
12 answers

Get DateTime.Now with milliseconds precision

How can I exactly construct a time stamp of actual time with milliseconds precision? I need something like 16.4.2013 9:48:00:123. Is this possible? I have an application, where I sample values 10 times per second, and I need to show them in a…
LuckyHK
  • 2,851
  • 2
  • 13
  • 6
257
votes
5 answers

Why are floating point numbers inaccurate?

Why do some numbers lose accuracy when stored as floating point numbers? For example, the decimal number 9.2 can be expressed exactly as a ratio of two decimal integers (92/10), both of which can be expressed exactly in binary (0b1011100/0b1010).…
mhlester
  • 22,781
  • 10
  • 52
  • 75
205
votes
11 answers

What's the difference between a single precision and double precision floating point operation?

What is the difference between a single precision floating point operation and double precision floating operation? I'm especially interested in practical terms in relation to video game consoles. For example, does the Nintendo 64 have a 64 bit…
meds
  • 21,699
  • 37
  • 163
  • 314
194
votes
3 answers

Why is a round-trip conversion via a string not safe for a double?

Recently I have had to serialize a double into text, and then get it back. The value seems to not be equivalent: double d1 = 0.84551240822557006; string s = d1.ToString("R"); double d2 = double.Parse(s); bool s1 = d1 == d2; // -> s1 is False But…
Philip Ding
  • 1,571
  • 2
  • 11
  • 11
193
votes
36 answers

Unexpected results when working with very big integers on interpreted languages

I am trying to get the sum of 1 + 2 + ... + 1000000000, but I'm getting funny results in PHP and Node.js. PHP $sum = 0; for($i = 0; $i <= 1000000000 ; $i++) { $sum += $i; } printf("%s", number_format($sum, 0, "", "")); //…
Baba
  • 94,024
  • 28
  • 166
  • 217
184
votes
24 answers

Retain precision with double in Java

public class doublePrecision { public static void main(String[] args) { double total = 0; total += 5.6; total += 5.8; System.out.println(total); } } The above code prints: 11.399999999999 How would I get…
Deinumite
  • 3,979
  • 3
  • 25
  • 22
164
votes
9 answers

Difference between toFixed() and toPrecision()?

I'm new to JavaScript and just discovered toFixed() and toPrecision() to round numbers. However, I can't figure out what the difference between the two is. What is the difference between number.toFixed() and number.toPrecision()?
Jessica
  • 1,651
  • 2
  • 10
  • 4
1
2 3
…
99 100