0

According to Is floating point math broken?, 0.1 would be rounded up to some value because 1/10 cannot be represented exactly. But my question is, how would a hardcoded decimal round up?

Does var a=0.1; round to the same value as var a=1/10;?

Or in general, does var a=x.yz; round to the same value as var a=xyz/100;?

ocomfd
  • 4,010
  • 2
  • 10
  • 19
  • So what have you tried and what have you discovered from that experiment? – Randy Casburn Feb 22 '18 at 02:05
  • You might want to have a look at http://www.ecma-international.org/ecma-262/6.0/#sec-tonumber-applied-to-the-string-type and http://www.ecma-international.org/ecma-262/6.0/#sec-literals-numeric-literals – Bergi Feb 22 '18 at 02:27
  • @RandyCasburn: Experiments do not provide definitive information about specifications. An experiment might tell you what one implementation does in one circumstance, but it will not tell whether it always does that, whether it is required to do that, or whether other implementations might behave differently. – Eric Postpischil Feb 22 '18 at 03:44
  • @EricPostpischil - and specifications do not tell you how any specific browser will implement the specification. But, apologize, I mistook your query for a question requiring an answer rather than a thesis. – Randy Casburn Feb 22 '18 at 05:45

1 Answers1

0

When using IEEE 754 arithmetic, normal operations operate as if they computed the exact mathematical result and then rounded it to the nearest representable value, using one of several choices for rounding rules, each of which is deterministic and rounds to a nearest value in some direction. With this behavior, the operation of converting the source text “x.yz” to a floating-point type and dividing xyz by 100 in the same type must have the same result.

The ECMA-262 specification, which standardizes JavaScript, says the Number type has values “representing the double-precision 64-bit format IEEE 754-2008 values as specified in the IEEE Standard for Binary Floating-Point Arithmetic” except that only one NaN is used. It further says “Mathematical operations such as addition, subtraction, negation, multiplication, division, and the mathematical functions defined later in this clause should always be understood as computing exact mathematical results on mathematical real numbers…” and later has text describing how such an exact mathematical result is used to choose a Number result, including “Choose the member of this set that is closest in value to x. If two values of the set are equally close,…” The phrasing and structure are a bit oblique, but I believe the intent is largely for operations to conform to IEEE 754.

Converting decimal numerals in strings (especially floating-point/scientific notation decimal, such as 1.2345e12) to floating-point has been a trouble spot for some programming languages or implementations of them. It is not always understood that this is a mathematical operation: A decimal numeral is being converted to a floating-point value. As a mathematical operation, it ought to be covered by the same rules: The result ought to be as if the operation were performed with exact mathematics and then rounded to the nearest representable value in a chosen direction, with a deterministic rule for resolving ties. The standard includes a note referring people to a classic paper on performing these conversions correctly, so my interpretation is that it intends these conversions to be performed correctly.

Given this, in JavaScript that conforms to ECMA-262, xy.z should have the same value as xyz/100. I would be open to correction in my interpretation by somebody more familiar with the ECMA-262 standard.

Eric Postpischil
  • 195,579
  • 13
  • 168
  • 312