0

When I run the following commands,

  (0..20).step(0.1) do |n|
     puts n
   end

I get the following return:

0.0
0.1
0.2
0.30000000000000004
0.4
0.5
0.6000000000000001
0.7000000000000001
0.8
0.9
1.0
1.1
1.2000000000000002
1.3
1.4000000000000001
1.5
1.6
1.7000000000000002
...

What is the best way to avoid this roundoff error?

Update: My question about why this occurs has been previously answered here in another question, Is floating point math broken?, but I did not immediately find that.

Michael Discenza
  • 3,240
  • 7
  • 30
  • 41
  • 5
    see [Is floating point math broken?](https://stackoverflow.com/questions/588004/is-floating-point-math-broken) – cremno Oct 12 '15 at 17:45
  • 4
    You may want to consider `BigDecimal` for such calculations: `require 'bigdecimal; (0..20).step(BigDecimal.new('0.1')) { |n| puts n.to_f }'` – spickermann Oct 12 '15 at 17:53
  • 1
    @spickermann, if you wish to step by one-third I don't think you can do it with `BigDecimal`. However, you could write: `step(Rational(1/3))`. For the example of a step of `0.1`, `(0..20).step(Rational(1, 10)) { |n| puts n.to_f }` produces the desired result. – Cary Swoveland Oct 12 '15 at 19:21
  • @Jordan, I've voted to re-open. The question cited concerns only float round-off error generally. Implied in this question is not only why this is happening by how it could be avoided. That's rather interesting: how can one step by a rational number without experiencing round-off error? Michael, could you please edit your question to ask how the round-off could be avoided? – Cary Swoveland Oct 12 '15 at 19:46
  • Michael, something doesn't compute. Given your academic background, you certainly know all about round-off error. – Cary Swoveland Oct 12 '15 at 19:49
  • @CarySwoveland I unfortunately did not learn about round off error because I took very little computer science. I read the the question and it makes sense now, but it wasn't immediately obvious what was going on when I saw the problem. I'm ended up using a the round function and that seemed like a satisfactory solution. Someone had left that as an answer but since deleted it. I'm happy to leave that as an answer below, or just delete the question here as I'm now embarrassed to have asked it. – Michael Discenza Oct 12 '15 at 21:00
  • Probably best to just delete the question, but consider posting another that asks what's the best way to deal with the round-off error. You might get some interesting answers. – Cary Swoveland Oct 12 '15 at 21:15

1 Answers1

1

You could cheat and avoid the stepping by 0.1 business:

(0..200).map { |n| n.to_f / 10 }

 => [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7,...]
yez
  • 2,368
  • 9
  • 15