using awk and printf not rounding correctly

Question

I have an issue where I'm using printf to round a float to the proper number of decimal points. I'm getting inconsistent results as shown below.

echo 104.45   |  awk '{printf "%.1f\n",$1}'
104.5                               <-- seem to be correct behaviour

echo 104.445  |  awk '{printf "%.2f\n",$1}'
104.44       (should be 104.45)     <-- seems to be INCORRECT behaviour

echo 104.4445 |  awk '{printf "%.3f\n",$1}'
104.445                             <-- seems to be correct behaviour

I've seen examples where float number in calculations may cause problems, but did not expect this with formatting.

For an even closer question, see [Python float round error 117.285 round to 117.28 not 117.29](http://stackoverflow.com/questions/9301690/python-float-round-error-117-285-round-to-117-28-not-117-29). — Sergey Salnikov, Dec 11 '16 at 07:47
Funny, what you consider correct I consider incorrect and vice-versa. In the context of formatting, I would expect no number representation issues whatsoever, just string shortening or padding. Only the second case has the expected behaviour for me. — bli, Dec 11 '16 at 07:53

user31264 · Answer 1 · 2016-12-11T08:14:30.223

The number 104.4445 cannot be represented exactly as a binary number. In other words, your computer doesn't know such a number.

# echo 104.4445 | awk '{printf "%.20f\n",$1}'
104.44450000000000500222

# echo 104.445 | awk '{printf "%.20f\n",$1}'
104.44499999999999317879

That's why the former is rounded to 104.445, while the latter is rounded to 104.44 .

The sjsam's answer is relevant only to numbers which can be represented exactly as a binary number, i. e. m/2**n , where m and n are integers and not too big. Changing ROUNDINGMODE to "A" has absolutely no effect on printing 104.45, 104.445, or 104.4445 :

# echo 104.4445  |  awk -v ROUNDMODE="A" '{printf "%.3f\n",$1}'
104.445
# echo 104.4445 | awk '{printf "%.3f\n",$1}'
104.445
# echo 104.445 | awk -v ROUNDMODE="A" '{printf "%.2f\n",$1}'
104.44
# echo 104.445 | awk '{printf "%.2f\n",$1}'
104.44

score 1 · Answer 2 · answered Dec 11 '16 at 08:05

I tried something analogous in Python and got similar results to you:

>>> round(104.445, 2)
104.44
>>> round(104.4445, 3)
104.445

This seems to be run-of-the-mill wonky floating point wonkiness, especially considering that the floating-point representation of 104.445 is less than the actual mathematical value of 104.445:

>>> 104.445 - 104.44
0.0049999999999954525
>>> 104.445 - 104.44 + 104.44
104.445

score 0 · Answer 3 · answered Dec 11 '16 at 08:24

I strongly suspect that the reason for this behavior has less to do with awk than with how computers store numbers. As user31264 states: "Your computer doesn't know such a number [as 104.4445]."

Here are the results of an experiment I just did with the JavaScript Scratchpad in Pale Moon Web browser:

(104.45).toFixed(55)
/*
104.4500000000000028421709430404007434844970703125000000000
*/

(104.445).toFixed(55)
/*
104.4449999999999931787897367030382156372070312500000000000
*/

(104.4445).toFixed(55)
/*
104.4445000000000050022208597511053085327148437500000000000
*/

In all probability, your awk interpreter is not dealing with the decimal numbers 104.45, etc., but rather with the "wonky" values shown here. Rounding the first, second, and third of these "wonky" values to, respectively, 1, 2, and 3 decimal places will give the same results as your awk interpreter is giving you.

using awk and printf not rounding correctly

3 Answers3