ScalaCheck: Gen.choose outputting values beyond a specified range

Question

I am using Scalacheck for finding defects, as a part of an assignment. Unbelievably perhaps, I am stuck as it is generating a pair of non-zero integers.

From my IntelliJ worksheet, ad verbatim:

import org.scalacheck._
    import Arbitrary._
    import Gen._
    import Prop._

    implicit lazy val genUnequalIntPairs = for {
      i <- Gen.choose(1,1000)
      j <- Gen.choose(i+1,1000)
      if (i < j)
    } yield (i,j)

    val kk = forAll (genUnequalIntPairs)  {
      case (x,y) => println("x =" + x + ", y =" + y)
        x == y
    }

kk.check

Because, I am explicitly mentioning the minimum of the chosen values to be non-zero, I should not see any zero inside the property, right? At least, that is my understanding. But this is what I see:

x =134, y =547
x =0, y =547
x =0, y =0
x =0, y =274
x =0, y =0
x =0, y =137
x =0, y =0
x =0, y =69
x =0, y =0
x =0, y =35
x =0, y =0
x =0, y =18
x =0, y =0
x =0, y =9
x =0, y =0
x =0, y =5
x =0, y =0
x =0, y =3
x =0, y =0
x =0, y =2
x =0, y =0
x =0, y =1
x =0, y =0
! Falsified after 0 passed tests.
> ARG_0: (0,1)
> ARG_0_ORIGINAL: (134,547)
res0: Unit = ()

From where are these zeroes coming? What I am missing? Perhaps, something obvious, but my eyes don't catch them.

I am using Scalcheck version 1.12.1

Seq("org.scalacheck" %% "scalacheck" % "1.12.1")

Any help, appreciated.

Update I: as suggested by @samar (I also found a reference to the same in gitbook), I tried to filter the generator with a suchThat. But not much luck (code below):

implicit lazy val genUnequalIntPairs = for {
  i <- Gen.choose(1,1000).suchThat(_ > 0)
  j <- Gen.choose(i+1,1000).suchThat(_ > 0)
  if (i < j)
} yield (i,j)

val kk = forAll (genUnequalIntPairs)  {
  case (x,y) => println("x =" + x + ", y =" + y)
    x == y
}

kk.check

I ran this code for about 6-7 times. This is one random output.

x =536, y =730
x =0, y =730
x =0, y =0
x =0, y =365
x =0, y =0
x =0, y =183
x =0, y =0
x =0, y =92
x =0, y =0
x =0, y =46
x =0, y =0
x =0, y =23
x =0, y =0
x =0, y =12
x =0, y =0
x =0, y =6
x =0, y =0
x =0, y =3
x =0, y =0
x =0, y =2
x =0, y =0
x =0, y =1
x =0, y =0
! Falsified after 0 passed tests.
> ARG_0: (0,1)
> ARG_0_ORIGINAL: (536,730)
res0: Unit = ()

Update II: Just to confirm, I have followed Eric's suggestions and have done this:

implicit lazy val genUnequalIntPairs = for {
  i <- Gen.choose(1,1000)
  j <- Gen.choose(i+1,1000)
  if (i != 0 && j != 0)
} yield {
  (i,j)
}

val kk = forAllNoShrink(genUnequalIntPairs)  {
  case (x,y) => println("x =" + x + ", y =" + y)
    x < y
}

kk.check

This works:

x =805, y =1000
x =742, y =926
x =772, y =919
x =219, y =686
x =999, y =1000
x =752, y =792
x =163, y =585
x =721, y =735
x =251, y =866
x =383, y =887
x =797, y =939
x =925, y =947
x =291, y =951
x =502, y =619
x =422, y =756
x =159, y =886
x =49, y =955
x =624, y =819
x =922, y =979
x =881, y =894
x =936, y =952
x =908, y =975
x =802, y =976
x =918, y =946
x =564, y =823
x =544, y =751
x =916, y =938
x =57, y =762
x =614, y =963
x =497, y =986
x =102, y =332
x =264, y =643
x =611, y =720
x =542, y =797
x =704, y =784
Output exceeds cutoff limit.

Update III: FWIW, I have found that by moving to Scalacheck version 1.13.1 (it was 1.12.x originally), I get the following behaviour:

implicit lazy val genUnequalIntPairs = for {
  i <- Gen.choose(1,1000)
  j <- Gen.choose(1,1000)
  if (i < j)
} yield {
  (i,j)
}

val kk = forAll(genUnequalIntPairs)  {
  case (x,y) => println("x =" + x + ", y =" + y)
    x < y
}

kk.check

produces

x =56, y =752
x =395, y =532
x =468, y =828
x =326, y =749
x =203, y =973
x =294, y =393
x =589, y =975
x =44, y =75
x =406, y =533
x =33, y =381
x =405, y =767
x =13, y =883
x =201, y =341
x =593, y =991
x =636, y =913
x =508, y =782
x =333, y =861
x =306, y =863
x =428, y =537
x =373, y =775
x =74, y =462
x =196, y =299
x =245, y =535
x =312, y =993
x =940, y =989
x =12, y =708

but by changing the condition to equality from less-than, this happens:

implicit lazy val genUnequalIntPairs = for {
  i <- Gen.choose(1,1000)
  j <- Gen.choose(1,1000)
  if (i < j)
} yield {
  (i,j)
}

val kk = forAll(genUnequalIntPairs)  {
  case (x,y) => println("x =" + x + ", y =" + y)
    x == y
}

kk.check

produces

x =370, y =585
x =0, y =585
x =0, y =0
x =0, y =293
x =0, y =0
x =0, y =147
x =0, y =0
x =0, y =74
x =0, y =0
x =0, y =37
x =0, y =0
x =0, y =19
x =0, y =0
x =0, y =10
x =0, y =0
x =0, y =5
x =0, y =0
x =0, y =3

Clearly, the condition being applied overrides the explicit instructions that I leave with the Generator. Isn't this somewhat non-intuitive? What do others think?

I think this is a bug and I've raised it: https://github.com/rickynils/scalacheck/issues/253 — , Aug 15 '16 at 17:47
@Samar, the range implies the predicate so really it shouldn't be necessary. By the way, it doesn't solve the issue as well. — , Aug 15 '16 at 18:11
@Mika'il: yeah i know its something fishy. Just wanted to see what happens if we add an extra guard. — Samar, Aug 15 '16 at 18:13
I have added two more testcases here: https://github.com/rickynils/scalacheck/issues/253 — Nirmalya, Aug 16 '16 at 02:20

score 5 · Accepted Answer · answered Aug 15 '16 at 18:03

5

It probably comes from the shrinking of properties. Try with forAllNoShrink instead.

answered Aug 15 '16 at 18:03

Eric

15,494
38
61

Just tried that - doesn't even get past the first test. – Aug 15 '16 at 18:10
Well your test is expecting to fail if x < y isn't it? – Eric Aug 15 '16 at 19:06
Actually it's not my code. Also commenting out that if condition still produces the problem. – Aug 15 '16 at 19:23
If you remove the condition and use `forAllNoShrink` you should not see any values with `0`. I still think you are experiencing this issue: http://stackoverflow.com/questions/20037900/scalacheck-wont-properly-report-the-failing-case. – Eric Aug 15 '16 at 19:28
Sorry, late in replying (early morning here). The code I presented is a sample; it just jogs the engine, doesn't really check anything meaningful. The idea is to prove that it disobeys the range of production I provide. My understanding is that 'Gen' should produce test-values, and 'Prop' should put them to, well, test! Why should Gen be dependent on the kind of condition applied later? Am I missing something? – Nirmalya Aug 16 '16 at 01:38
I have updated the original post, to confirm that by using 'forAllNoShrink' (as suggested by @Eric, thanks), I can get past the oirginal problem. However, I am still not being able to explain, why a Generator's behaviour be dependent upon the property it is supposed to help test. – Nirmalya Aug 16 '16 at 02:28
This indeed is a bug. "Auto shrinking of properties" - finding smallest input that exhibits erroneous behaviour: should not shrink them below a an explicitly chosen range. – Samar Aug 16 '16 at 06:26
1

Thanks @Samar, for confirming my doubt. Also, I am accepting. Eric's suggestion, because it allows one to get past the irritant. Github ticket by mikai'l contains all the testcases I have shared here, so far. – Nirmalya Aug 16 '16 at 06:49
Ricky (https://github.com/rickynils) has added a clear explanation behind the behaviour of Scalacheck here: https://github.com/rickynils/scalacheck/issues/253 . Many thanks, Ricky @rickard-nillson – Nirmalya Aug 18 '16 at 05:37

ScalaCheck: Gen.choose outputting values beyond a specified range

1 Answers1