4

I've come across a curious issue in one of my unit tests where I'm getting unexpected rounding results in JavaScript:

(2.005).toFixed(2)
// produces "2.00"

(2.00501).toFixed(2)
// produces "2.01"

Initially I suspected this was a Webkit only issue but it repros in Gecko which implies to me that it is an expected side effect of either ECMA-262 or IEEE-754. I'm assuming the binary representation of 2.005 is ever so slightly less? Or does ECMA-262 specify a round-to-even methodology for toFixed?

Anyone care to shed some insight as to what is happening under the hood just to give me peace of mind?

Update: thanks for the comments.

I should add, one of the things that made me a little nervous was the comments found in a quick search in Webkit dtoa.cpp which seemed to imply that there were multiple paths to rounding and the devs weren't really sure how it worked, including a related FIXME:

https://trac.webkit.org/browser/trunk/Source/WTF/wtf/dtoa.cpp#L1110

Also, not that it means much but IE9 rounds it as I expected, implying that it either isn't part of ECMA-262 or they have a bug.

mckamey
  • 17,359
  • 16
  • 83
  • 116
  • 1
    Most likely `2.005` is actually a `2.004999999...` in binary floating-point. – Mysticial Nov 06 '12 at 22:26
  • 2
    2.00499999999999989341858963598497211933135986328125 – Daniel Fischer Nov 06 '12 at 22:27
  • Perhaps [have a look here](http://stackoverflow.com/questions/1458633/elegant-workaround-for-javascript-floating-point-number-problem) There's a lot of links to articles on the issue, most important link being [the possible duplicate](http://stackoverflow.com/questions/588004/is-javascripts-math-broken) – Elias Van Ootegem Nov 06 '12 at 22:31
  • @EliasVanOotegem ironically, I was using `toFixed` to get around one of these kinds of quirks when I discovered this! – mckamey Nov 06 '12 at 22:38
  • @McKAMEY: `toFixed` isn't very reliable, it has some X-browser issues, for example. Thankfully, someone went through the trouble of writing a workaround, [but I'm not sure if you'd want to use it](http://jibbering.com/faq/#formatNumber) – Elias Van Ootegem Nov 06 '12 at 22:44
  • @DanielFischer how did you calc the exact representation? Would be helpful to do at times. – mckamey Nov 06 '12 at 23:32
  • 1
    I defined [a type](http://hackage.haskell.org/packages/archive/floatshow/0.2.3/doc/html/Text-FShow-RealFloat.html#v:FD) in my Haskell library for that purpose. You can also get the full representation from gcc if you ask for a high enough precision (`%.52f` is enough for values `>= 1`), but since I always have a ghci running, it's faster to ask that. – Daniel Fischer Nov 06 '12 at 23:44

1 Answers1

9

If the specification hasn't changed since Rev. 6 of the ECMA 262 draft (edition 5.1, March 2011), (2.005).toFixed(2) must return the string "2.00", since a "Number value" is a

primitive value corresponding to a double-precision 64-bit binary format IEEE 754 value

and the interpretation of numeric literals is specified in 7.8.3 and 8.5 to conform to IEEE 754 "round to nearest" mode (with ties rounded to even significand), which for 2.005 results in the value

x = 4514858626438922 * 2^(-51) = 2.00499999999999989341858963598497211933135986328125

In section 15.7.4.5 which deals with toFixed, the relevant step 8. a. is:

Let n be an integer for which the exact mathematical value of n ÷ 10f – x is as close to zero as possible. If there are two such n, pick the larger n.

and 2.00 - x is closer to zero than 2.01 - x, so n must be 200 here. The conversion to a string proceeds then in the natural way.

Also, not that it means much but IE9 rounds it as I expected, implying that it either isn't part of ECMA-262 or they have a bug.

A bug. Maybe they tried to go the easy way and multiply with 10^digits and round. x*100 is exactly 200.5, so that would produce a string of "2.01".

Daniel Fischer
  • 181,706
  • 17
  • 308
  • 431