ScalaCheck, different behaviour when calling Test.check or Test.checkProperties

Question

I have the following code that uses ScalaCheck properties to test some class

package quickcheck.tests

import org.scalacheck.Arbitrary._
import org.scalacheck.Gen._
import org.scalacheck.Prop._
import org.scalacheck._
import org.scalacheck.util.ConsoleReporter

class Heap
{
}

object Heap
{
  val empty = new Heap
  def insert(i: Int, h: Heap) = new Heap
}

class TestCheck extends Properties("Heap")
{
  lazy val genHeap: Gen[Heap] =
  {
    def sizedHeap(size: Int): Gen[Heap] =
    {
      if (size <= 0)
        Heap.empty
      else
        for
        (
          i <- arbitrary[Int] suchThat(_ > Int.MinValue);
          s <- choose(0, size);
          h <- sizedHeap(s)
        )
          yield
            Heap.insert(i, h)
    }

    Gen.sized[Heap](sizedHeap)
  }

  implicit lazy val arbHeap: Arbitrary[Heap] = Arbitrary(genHeap)

  property("test1") = forAll
  {
    (h: Heap) => true
  }

  property("test2") = forAll
  {
    (h1: Heap, h2: Heap, n: Int) => true
  }
}

object MyTest extends App
{
  println("*** TEST 1")
  val checkHeap = new TestCheck
  Test.checkProperties(Test.Parameters.default.withTestCallback(ConsoleReporter(1)), 
    checkHeap)

  println("*** TEST 2")
  val checkHeap2 = new TestCheck
  checkHeap2.check

  println("*** TEST 3")
  val checkHeap3 = new TestCheck
  Test.check(Test.Parameters.default.withTestCallback(ConsoleReporter(1)),     checkHeap)
}

If I run it thru the ScalaCheck Test class I get different results if I use method Test.checkProperties or method Test.check.

This is the output I get:

*** TEST 1
+ Heap.test1: OK, passed 100 tests.
+ Heap.test2: OK, passed 100 tests.
*** TEST 2
+ Heap.test1: OK, passed 100 tests.
+ Heap.test2: OK, passed 100 tests.
*** TEST 3
! Gave up after only 40 passed tests. 202 tests were discarded.

My questions is why TEST1 gives different result than TEST3.

If I remove the suchThat filter and leave the for statement in the sizeHead method as this:

for
(
  i <- arbitrary[Int]
  s <- choose(0, size);
  h <- sizedHeap(s)
)
  yield
    Heap.insert(i, h)

I get the following result:

*** TEST 1
+ Heap.test1: OK, passed 100 tests.
+ Heap.test2: OK, passed 100 tests.
*** TEST 2
+ Heap.test1: OK, passed 100 tests.
+ Heap.test2: OK, passed 100 tests.
*** TEST 3
+ OK, passed 100 tests.

Is this a bug or it is the correct behaviour? Keep in mind that ScalaTest Checkers.check uses Test.check.

score 1 · Accepted Answer · 2016-09-20T10:39:31.487

1

You get different results because you're doing different things. Your first two tests—which are effectively the same—check all properties separately, whereas your third one tests all properties as if they were a single property.

Take a look at the signatures of Test.check and Test.checkProperties: The former takes a single Prop, whereas the latter takes a Properties container.

In scalacheck 1.12 Properties inherits from Prop; if you pass Properties as Prop you get a new property that tests whether all contained properties hold. The consequence is that you test all your TestCheck properties with a single check configuration (i.e. generators, iteration limits, etc.), and naturally that check configuration is exhausted at some point.

So yes, that's expected behaviour. Of course, it's absolutely confusing, hence scalacheck 1.13 removed this feature: Properties does not inherit from Prop anymore; your example would not compile on scalacheck 1.13 anymore.

edited Sep 20 '16 at 10:39

answered Sep 20 '16 at 09:56

Thank you for your response. Yes I noticed that I was doing different things, what I do not understand is why I am getting different results for two checks that are supposed to do the same thing that is to check all the properties. Regarding the typo, you are right,, when reformatting the code, that I tested before posting, I deleted by error a right parenthesis, this produced the syntax error. I beg you pardon. But unfortunately I am pretty sure that this will happen again if I cannot copy and paste code without modifying it to reformat. – Josep M Beleta Sep 20 '16 at 11:26
By the way I edited the question to correct the typo. – Josep M Beleta Sep 20 '16 at 11:28
Yes, thanks again, now things are little bit clearer – Josep M Beleta Sep 20 '16 at 11:32
In the dark side again. It seems that ScalaTest latest realeases do not support ScalaCheck 1.13 http://stackoverflow.com/questions/35312410/java-lang-incompatibleclasschangeerror-implementing-class-with-scalacheck-and-s. This Scala world seems so dilettante! – Josep M Beleta Sep 20 '16 at 12:38
Yes that's know—but do you need to update scalacheck? scalatest works just fine with 1.12, we're using it every day, and the changes in 1.13 are unlikely to affect you much. If you're using scalatest's property check support you won't be affected by the behaviour you've demonstrated in your question. In any case, I find your last sentence very unfair and offensive. It's not unusual for updates to tickle downstream slowly, and scalacheck is a volunteer project. I'm sure scalatest would accept your pull request. – Sep 20 '16 at 13:00
I try to switch to ScalaCheck 1.13 because of your comment. My last sentence has nothing to do with the nice people at ScalaTest. It simply reflects the fact that assuring binary compatibility is not at so high priority for Scala SDK developers as for Java SDK ones. But I beg you pardon again if you found it so offensive – Josep M Beleta Sep 20 '16 at 14:04
@jmbeleta I'm sorry, there may be a misunderstanding: scalacheck 1.13 removing the cause of the confusing behaviour doesn't imply that scalacheck 1.12 is broken. It works fine with scalatest, and scalatest doesn't use "Test.check" the way you did, so it won't suffer from any problem here. – Sep 20 '16 at 14:13

ScalaCheck, different behaviour when calling Test.check or Test.checkProperties

1 Answers1