Why is Ruby adding precision/digits when calculating a mod?

Question

I need to make some calculations with coordinates and came across this (at least for me) weird behaviour. Can someone explain why this happens?

$ long
=> 49.0126760222489
$ long % long.floor
=> 0.012676022248896857

I was expecting the last line to evaluate to just the digits after the period, from long, but instead there is an additional 6857 tacked onto the end.

Why is it adding digits?

Where is the information coming from?

This is an artefact of the way floating point math works about everywhere. The exact same effect can be shown in e.g. Python and Javascript. Smaller numbers can be represented with higher precision than larger numbers. — Holger Just, Oct 05 '16 at 13:54
Note that the discrepancy is at the scale of [microns](https://en.wikipedia.org/wiki/Micrometre) (assuming these are terrestrial coordinates) — Stefan, Oct 05 '16 at 16:02

potashin · Answer 1 · 2016-10-05T23:15:40.833

3

It seems that it has nothing to do with Ruby itself, but:

floating point numbers that in general suffer from this type of rounding error, because they’re limited to x number of bytes, and, in general, cannot store decimal number perfectly.

Documentation

From the docs above one can also get a couple of options for getting around the native float behaviour. Documentation above can be considered as an official one as it is provided by ruby-doc.org.

edited Oct 05 '16 at 23:15

answered Oct 05 '16 at 13:32

potashin

44,205
11
83
107

3

This is not a "more precise" result, this is a *less* precise result. It would be more precise if it added `0000` onto the end, adding `6857` just makes the result *wrong* by a small margin. – user229044 Oct 05 '16 at 13:32
converting `long.floor` Fixnum value seems irrelevant to me, since the `long` itself is floating point already. I find no suprise in float%fixnum resulting in a float. I suppose we simply see float's imprecions. See i.e. `0.0126760222489 % 1.0` and `1.0126760222489 % 1.0` – quetzalcoatl Oct 05 '16 at 13:39

score 2 · Accepted Answer · answered Oct 05 '16 at 13:55

I suppose we simply see float's imprecions. See i.e. 0.0126760222489 % 1.0 and 1.0126760222489 % 1.0. You'd think the result should be the same, but no - IEEE754 floats/doubles don't guarantee perfect results, and by default these are used in Ruby for storing floating point values.

It's even somewhat shown in the docs

6543.21.modulo(137)      #=> 104.21
6543.21.modulo(137.24)   #=> 92.9299999999996

you can see that the second result has a small error. Actually, on Ruby 2.3.1 I ran the first line and got:

pry(main)> 6543.21.modulo(137)
=> 104.21000000000004

It's not necessarily related to modulo, and not always visible:

[30] pry(main)> 10.0126760222489 - 0.0
=> 10.0126760222489
[31] pry(main)> 10.0126760222489 - 1.0
=> 9.0126760222489
[32] pry(main)> 10.0126760222489 - 2.0
=> 8.0126760222489
[33] pry(main)> 10.0126760222489 - 3.0
=> 7.0126760222489
[34] pry(main)> 10.0126760222489 - 4.0
=> 6.0126760222489
[35] pry(main)> 10.0126760222489 - 5.0
=> 5.0126760222489
[36] pry(main)> 10.0126760222489 - 6.0
=> 4.0126760222489
[37] pry(main)> 10.0126760222489 - 7.0
=> 3.0126760222489004

Every piece of software that uses standard floats needs to account for these small errors. If you cannot handle that for some reason, then you could use bigdecimal (should be already included in your Ruby), fixed-point, or some similar numeric library

require 'bigdecimal'

BigDecimal('6543.21').modulo(137).to_s
=> "0.10421E3"

BigDecimal('6543.21').modulo(137).to_f
=> 104.21

Keep in mind that 'bigdecimal' may be slower and may use more memory.

Great, thanks for answering despite the fact that it's been asked ( and answered) before — Flip, Oct 05 '16 at 14:03

Why is Ruby adding precision/digits when calculating a mod?

2 Answers2