What's the difference between shouldBe and shouldBe exactly in KotlinTest?

Question

This is a test code using KotlinTest 1.3.5.

val expect = 0.1
val actual: Double = getSomeDoubleValue() 
actual shouldBe expect

and this warning was printed when the code was run.

[WARN] When comparing doubles consider using tolerance, eg: a shouldBe b plusOrMinus c

In this case, I didn't want to use plusOrMinus. So, I fixed the code to

val expect = 0.1
val actual: Double = getSomeDoubleValue() 
actual shouldBe exactly(expect)

Now, there is no warning.
However, I'd like to know the difference between shouldBe and shouldBe exactly. What is it?

https://github.com/kotlintest/kotlintest#matchers – Jehof Nov 15 '16 at 13:45 — Jehof, Nov 15 '16 at 13:45

score 6 · Accepted Answer · edited May 23 '17 at 10:30

According to the current sources:

infix fun Double.shouldBe(other: Double): Unit = ToleranceMatcher(other, 0.0).test(this)

where ToleranceMatcher is

class ToleranceMatcher(val expected: Double, val tolerance: Double) : Matcher<Double> {

  override fun test(value: Double) {
    if (tolerance == 0.0)
      println("[WARN] When comparing doubles consider using tolerance, eg: a shouldBe b plusOrMinus c")
    val diff = Math.abs(value - expected)
    if (diff > tolerance)
      throw AssertionError("$value is not equal to $expected")
  }

  infix fun plusOrMinus(tolerance: Double): ToleranceMatcher = ToleranceMatcher(expected, tolerance)
}

So, matching d shouldBe e will compare the doubles exactly without any tolerance (a - b will never give 0 for different doubles) and print the warning:

[WARN] When comparing doubles consider using tolerance, eg: a shouldBe b plusOrMinus c

Whereas exactly(d) is defined as

fun exactly(d: Double): Matcher<Double> = object : Matcher<Double> {
    override fun test(value: Double) {
      if (value != d)
        throw AssertionError("$value is not equal to expected value $d")
    }
}

So that it will do the same, though without any warning.

I guess, the meaning of this warning is to encourage developers to specify it explicitly that doubles are compared exactly, or else specify the tolerance, because even the same arithmetics done in different order might produce different results with doubles.

Actually the warning is slightly misleading and it must be `a shouldBe (b plusOrMinus c)`. See https://github.com/kotlintest/kotlintest/issues/259 — Holger Brandl, Mar 14 '18 at 10:44

score 4 · Answer 2 · answered Nov 15 '16 at 13:51

Comparing floating point numbers, especially ones that are calculated, is always subject to rounding error due to the fact that some floating point numbers might not be representable give the way they are converted to binary. So when testing you usually have to give a value for what you consider an acceptable rounding error.

In KotlinTest this is specified using the plusOrMinus specifier.

If there are cases when you know that the number should be exactly a given value (e.g. it's initialised to 0 or 10 or whatever) then use the exactly specifier. Presumably this does something behind the scenes to specify a really small error value or compares the doubles in some other way.

To assert that a double is exactly equal to another double use d shouldBe exactly(e)
To assert that a double is equal within some tolerance range, use d shouldBe (e plusOrMinus y)

Source

If you just use shouldBe on it's own then the system will just do a simple if a == b type test which will expose your test to the above mentioned rounding errors.

What's the difference between shouldBe and shouldBe exactly in KotlinTest?

2 Answers2