4

I'm working with kotlin + Kotest property testing and trying to test all permutations of 2 parameters with list generators like this:

"Some test"{
        forAll(4 ,
                Exhaustive.collection(listOf(
                        "a",
                        "b")),
                Exhaustive.collection(listOf(
                        "1",
                        "2"))
        { begins_with, contains ->
            println("$begins_with:$contains")
            ... some validation code...
        }

I hoped that using the exhaustive generator would generate them in such a way that with 4 iterations all possible permutations would be covered. Like this:

a:1
b:2
a:2
b:1

Instead, exhaustive generators always go in the listed order like this:

a:1
b:2
a:1
b:2

which means I'm testing the same case multiple times.

I've tried switching some generators to Arbs, and that DOES switch the order up, but not optimally. To increase the likelihood of hitting all cases I have to test more than I would if the right order was used.

I've also considered listing the same element multiple times like this

"Some test"{
        forAll(4 ,
                Exhaustive.collection(listOf(
                        "a",
                        "b")),
                Exhaustive.collection(listOf(
                        "1",
                        "1",
                        "2",
                        "2"))
        { begins_with, contains ->
            println("$begins_with:$contains")
            ... some validation code...
        }

but that doesn't seem sustainable, especially when I want to add more parameters or values later.

Is there a way to generate the exhaustive permutations, rather than just continuing to loop over each list?

janicedn
  • 53
  • 5
  • You could try jqwik’s exhaustive generation feature, which does what you want; https://jqwik.net/docs/current/user-guide.html#exhaustive-generation – johanneslink Jul 24 '20 at 04:57
  • 1
    Kotest 4.2 (not yet released) has automatic cross product support for Exhaustives. – sksamuel Jul 26 '20 at 19:08
  • 1
    Kotest 4.2 has been released which works in the exact way you wanted and described above. – sksamuel Aug 24 '20 at 00:59
  • 1
    Thanks @sksamuel for the update! I will check it out. In addition to the other suggestion I'm also checking out propCheck/ArrowCheck. Seems like there's a lot of cools stuff in this space. – janicedn Aug 24 '20 at 03:37

1 Answers1

5

It seems that you want to merge two exhaustives. Currently, one way of doing this is using Exhaustive.times, which will produce the cross product of two exhaustives:

Exhaustive.collection(listOf("a", "b"))
.times(Exhaustive.collection(listOf("1", "2"))
.checkAll(4) { (begins, contains) ->
    println("$begins:$contains")
}

However, if your usecase is similar to the one you presented with your question, I would suggest Kotest Inspectors instead:

listOf("a", "b").forAll { first ->
    listOf("1", "2").forAll { second ->
        println("$first:$second")
    }
}
LeoColman
  • 6,950
  • 7
  • 34
  • 63