Ruby rounding and formatting of decimal numbers

Question

I have looked through: Why can't decimal numbers be represented exactly in binary? and Why Are Floating Point Numbers Inaccurate?

My question is when I round and format floating point numbers using ruby, the results are still different than the result of doing the math buy hand. Below is an example of this:

Edit 2

[28] pry(main)> #####################################
[29] pry(main)> # test one
[30] pry(main)> #####################################
[31] pry(main)> foo = (6.0135 * (650000 / 1000)) 
             => 3908.7749999999996
[32] pry(main)> foo = '%.2f' % foo.round(2)
             => "3908.77"
[33] pry(main)> # should be 3908.78

[36] pry(main)> #####################################
[37] pry(main)> # test two
[38] pry(main)> #####################################
[39] pry(main)> foo = 650000 / 1000
             => 650
[40] pry(main)> foo = foo * 6.0135
             => 3908.7749999999996
[41] pry(main)> foo = '%.2f' % foo.round(2) # should be 3908.78
             => "3908.77"

[44] pry(main)> #####################################
[45] pry(main)> # test three 
[46] pry(main)> #####################################
[47] pry(main)> foo = 650000 / 1000
             => 650
[48] pry(main)> foo = foo * 6.0135
             => 3908.7749999999996
[49] pry(main)> foo = foo.round(2) # should be 3908.78
=> 3908.77

[52] pry(main)> #####################################
[53] pry(main)> # test four - The result of test four is expected
[54] pry(main)> #####################################
[55] pry(main)> foo = 650000 / 1000
             => 650
[56] pry(main)> foo = foo * 6.0135
             => 3908.7749999999996
[57] pry(main)> foo = '%.2f' % foo
             => "3908.77"

[58] pry(main)> #####################################
[59] pry(main)> # test five
[60] pry(main)> #####################################
[61] pry(main)> foo = 650000 / 1000
             => 650
[62] pry(main)> foo = foo * 6.0135
             => 3908.7749999999996
[63] pry(main)> foo = foo.round(5) 
             => 3908.775

Test 1: This is the normal formula I use in my library that is giving me the issue.

Test 2: My thought here was that maybe doing both operations in one assignment was causing some problems

Test 3: Well since '%.2f' % has been known to cause some rounding issues where it truncates things, maybe this is the issues.

Test 4: Since '%.2f' % wasn't the problem maybe the .round(2) is causing the problem.

Edit 2: Test 5: If I expand the digits that round is looking at I get a number that I could work with (ie. round(2) that number again). But this seems like a wonky solution that works for a limited case and not in general

I don't really care that the floating point is inaccurate. My question is how can I round this number correctly so that I get the correct answer (the answer you would get if you performed the operation by hand).

Also in general is there a best practice for rounding to correct for dosing point errors? I know it is to use two integers to represent floating point numbers, but that seems like a hassle.

Thank you.

Edit 1

If you use .round(5) instead of .round(2) you would get a more reasonable answer when comparing it to what the answer should be. Is there any drawbacks to increasing the number inside of the round?

This round is correct. 3908.77499999.. will always be 3908.77 because it is indeed smaller than 3908.775. You always round the the total value, you dont start on the last digit round that and then go forth.... — Doktor OSwaldo, Feb 21 '17 at 08:06
Aside from the fact that the actual answer should be 3908.78. If you use `.round(5)` you get 3908.78 — alex_milhouse, Feb 21 '17 at 08:24
To be clear, the downvote is not from me, but it honestly should be 3908.77 with .round(2). And .round(5) should give you 3908.775, and it does so in my rails console — Doktor OSwaldo, Feb 21 '17 at 08:33
@DoktorOSwaldo Yeah no worries! Is using .round(5) better? I will add that to the question as well. — alex_milhouse, Feb 21 '17 at 08:41
@DavidAldridge it was more that I didn't understand how to use the round function to correct the issue. — alex_milhouse, Feb 21 '17 at 09:07
Please, post the code your are having trouble with. Not a picture of the code, *the code*. — Jörg W Mittag, Feb 21 '17 at 09:22
@JörgWMittag Thanks for the feedback! I updated the question. — alex_milhouse, Feb 21 '17 at 19:29

Stefan · Answer 1 · 2017-02-21T08:59:01.993

5

round is working as expected. You get the wrong result, because your input is already flawed. The floating point number 6.0135 is actually:

6.01349999999999962341235004714690148830413818359375

Multiplying this number by 650 makes the error worse. You get a result that is closer to 3908.77 than to 3908.78:

foo = 6.0135 * 650
#=> 3908.7749999999996

(foo - 3908.77).abs
#=> 0.004999999999654392

(foo - 3908.78).abs
#=> 0.005000000000563887

To get the correct result, you could use something like this:

foo = (6.0135 * 10000).round  * 0.065
#=> 3908.775

foo.round(2)
#=> 3908.78

Or you could use BigDecimal to avoid floating point errors in the first place:

require 'bigdecimal'
foo = BigDecimal('6.0135') * 650
foo.to_s('F')
#=> "3908.775"

foo.round(2).to_s('F')
#=> "3908.78"

edited Feb 21 '17 at 08:59

answered Feb 21 '17 at 08:50

Stefan

109,145
14
143
218

Thank you. Are there major draw backs to the bigdecimal, other than some performance hit? I intend to do this kind of math 25 times per request, will this change have much of an affect on performance of the request? – alex_milhouse Feb 21 '17 at 09:31
1

@alex_milhouse a request-response cycle in Rails is not that lightweight. I would be surprised if 25 calculations would affect it noticeably. – Stefan Feb 21 '17 at 09:45
great. Thank you, this was exactly what I needed. – alex_milhouse Feb 21 '17 at 18:57

alex_milhouse · Accepted Answer · 2017-02-21T08:39:32.447

1

'%.2f' % - truncates the number and fails to round correctly

.round(2) - doesn't look enough decimal places back to round correctly. So if you make the number inside of the greater than 2 the issue of rounding is gone.

Edit 1

There are 5 ways to round a number according to the IEEE:

to nearest, ties to even
to nearest, ties away from zero
toward 0
toward +∞
toward −∞

In this case it isn't so much about how I am rounding but it is more about the number of digits after the decimal point that need to be considered. Typically a person would need to worry about precision. Since this error is created by the computer, I am assuming that increasing the number of values after the decimal point that are considered is better and more accurate.

edited Feb 21 '17 at 08:39

answered Feb 21 '17 at 07:56

alex_milhouse

891
1
13
31

1

I'd think that you'd need to justify these statements by citing the IEEE floating point standards before this would qualify as an answer. – mu is too short Feb 21 '17 at 08:13
Yeah @muistooshort and lukasz I am actually having second thoughts about how I "solved" this. While this gives me the answer I wanted for this specific case. Are there other issues with doing this. – alex_milhouse Feb 21 '17 at 08:15

score 0 · Answer 3 · answered Feb 21 '17 at 08:40

I tested it all on my Rails Console, and can not see any strange behaviour:

>> foo = 650000 / 1000
=> 650
>> foo = foo * 6.0135
=> 3908.7749999999996
>> foo.round(2)
=> 3908.77
>> foo.round(3)
=> 3908.775
>> foo.round(5)
=> 3908.775
>> afoo = '%.2f' % foo
=> "3908.77"
>> afoo = '%.5f' % foo
=> "3908.77500"

3908.7749999999996 is smaller than 3908.775 and will be rounded down to 3908.77 if you use .round(2). .round(5) results as expected in 3908.775 which can also be reproduced witht he google calculator. Why do you think this should be incorrect ? I would really like to understand that.

Ruby rounding and formatting of decimal numbers

3 Answers3