Kotest breaks gradle's :test task with java.lang.IllegalArgumentException: Received a failure event for test with unknown id

Question

I'm getting started with unit tests in Kotlin using Kotest. I use the following technologies that integrate somehow with Kotest:

Kotest itself
Kotlin / JVM
Gradle
Allure
Pitest
IntelliJ IDEA Plugin "Kotest"

In gradle, I included the following dependencies:

io.kotest:kotest-runner-junit5:$kotest_version: Kotest Framework
io.kotest:kotest-assertions-core:$kotest_version: Kotest Core JVM Assertions
io.kotest:kotest-property:$kotest_version: Kotest Property Test
io.kotest:kotest-extensions-allure:$kotest_version: Data collection for Allure
io.kotest:kotest-plugins-pitest:$kotest_version: Plugin for Pitest

The problem is that now, when I run my tests via the gradle :test task, I get the following exception:

java.lang.IllegalArgumentException: Received a failure event for test with unknown id '2.27'. Registered test ids: '[2.1, :test, 2.25]'

The unknown id / registered ids differ in every test run. Actually there are a lot of errors appearing, but that one is the last one to appear. Below is the full gradle output (shortened internal calls):

Testing started at 18:39 ...

> Task :kaptGenerateStubsKotlin UP-TO-DATE
> Task :kaptKotlin UP-TO-DATE
> Task :compileKotlin UP-TO-DATE
> Task :compileJava NO-SOURCE
> Task :processResources UP-TO-DATE
> Task :classes UP-TO-DATE
> Task :kaptGenerateStubsTestKotlin UP-TO-DATE
> Task :kaptTestKotlin UP-TO-DATE
> Task :compileTestKotlin UP-TO-DATE
> Task :preTest
> Task :compileTestJava NO-SOURCE
> Task :processTestResources UP-TO-DATE
> Task :testClasses UP-TO-DATE
> Task :test
~~~ Kotest Configuration ~~~
-> Parallelization factor: 1
-> Default test timeout: 600000ms
-> Default test order: Sequential
-> Default isolation mode: SingleInstance
-> Global soft assertations: False
-> Write spec failure file: False
-> Fail on ignored tests: False
-> Spec execution order: SpecExecutionOrder
-> Extensions
  - io.kotest.engine.extensions.SystemPropertyTagExtension
  - io.kotest.core.extensions.RuntimeTagExtension
  - io.kotest.engine.extensions.RuntimeTagExpressionExtension
-> Listeners
  - io.kotest.extensions.allure.AllureTestReporter
  - class io.kotest.engine.config.LoadConfigFromClasspathKt$toDetectedConfig$beforeAfterAllListener$1

Nov. 04, 2020 6:39:48 NACHM. org.junit.platform.launcher.core.TestExecutionListenerRegistry lambda$notifyEach$1
WARNING: TestExecutionListener [org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestExecutionListener] threw exception for method: executionFinished(TestIdentifier [uniqueId = '[engine:kotest]/[spec:com.example.core.common.JSONFileTests]/[test:Writing to and reading from file should result in the original value]', parentId = '[engine:kotest]/[spec:com.example.core.common.JSONFileTests]', displayName = 'Writing to and reading from file should result in the original value', legacyReportingName = 'Writing to and reading from file should result in the original value', source = FileSource [file = D:\dev\Minecraft Server Watcher\msw-server\JSONFileTests.kt, filePosition = FilePosition [line = 40, column = -1]], tags = [], type = TEST], TestExecutionResult [status = FAILED, throwable = java.time.temporal.UnsupportedTemporalTypeException: Unsupported field: OffsetSeconds])
java.lang.AssertionError <3 internal calls>
    at jdk.internal.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) <11 internal calls>
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1540) <17 internal calls>
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
    at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
    at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:84)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
    at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
    at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source) <5 internal calls>

Nov. 04, 2020 6:39:48 NACHM. org.junit.platform.launcher.core.TestExecutionListenerRegistry lambda$notifyEach$1
WARNING: TestExecutionListener [org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestExecutionListener] threw exception for method: executionFinished(TestIdentifier [uniqueId = '[engine:kotest]/[spec:com.example.core.common.JSONFileTests]/[test:Writing to and reading from file should result in the original value]', parentId = '[engine:kotest]/[spec:com.example.core.common.JSONFileTests]', displayName = 'Writing to and reading from file should result in the original value', legacyReportingName = 'Writing to and reading from file should result in the original value', source = FileSource [file = D:\dev\Minecraft Server Watcher\msw-server\JSONFileTests.kt, filePosition = FilePosition [line = 40, column = -1]], tags = [], type = TEST], TestExecutionResult [status = FAILED, throwable = java.time.temporal.UnsupportedTemporalTypeException: Unsupported field: OffsetSeconds])
java.lang.AssertionError <3 internal calls>
    at jdk.internal.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) <11 internal calls>
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1540) <17 internal calls>
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
    at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
    at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:84)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
    at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
    at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source) <5 internal calls>
    at java.base/java.lang.Thread.run(Thread.java:834)

Nov. 04, 2020 6:39:48 NACHM. org.junit.platform.launcher.core.TestExecutionListenerRegistry lambda$notifyEach$1
WARNING: TestExecutionListener [org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestExecutionListener] threw exception for method: executionFinished(TestIdentifier [uniqueId = '[engine:kotest]/[spec:com.example.core.common.JSONFileTests]/[test:Writing to and reading from file should result in the original value]', parentId = '[engine:kotest]/[spec:com.example.core.common.JSONFileTests]', displayName = 'Writing to and reading from file should result in the original value', legacyReportingName = 'Writing to and reading from file should result in the original value', source = FileSource [file = D:\dev\Minecraft Server Watcher\msw-server\JSONFileTests.kt, filePosition = FilePosition [line = 40, column = -1]], tags = [], type = TEST], TestExecutionResult [status = SUCCESSFUL, throwable = null])
java.lang.AssertionError <3 internal calls>
    at jdk.internal.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) <11 internal calls>
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1540) <17 internal calls>
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
    at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
    at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:84)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
    at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
    at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source) <5 internal calls>
    at java.base/java.lang.Thread.run(Thread.java:834)

Nov. 04, 2020 6:39:48 NACHM. org.junit.platform.launcher.core.TestExecutionListenerRegistry lambda$notifyEach$1
WARNING: TestExecutionListener [org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestExecutionListener] threw exception for method: executionFinished(TestIdentifier [uniqueId = '[engine:kotest]/[spec:com.example.core.common.JSONFileTests]/[test:Writing to and reading from file should result in the original value]', parentId = '[engine:kotest]/[spec:com.example.core.common.JSONFileTests]', displayName = 'Writing to and reading from file should result in the original value', legacyReportingName = 'Writing to and reading from file should result in the original value', source = FileSource [file = D:\dev\Minecraft Server Watcher\msw-server\JSONFileTests.kt, filePosition = FilePosition [line = 40, column = -1]], tags = [], type = TEST], TestExecutionResult [status = SUCCESSFUL, throwable = null])
java.lang.AssertionError <3 internal calls>
    at jdk.internal.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) <11 internal calls>
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1540) <17 internal calls>
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
    at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
    at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:84)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
    at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
    at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source) <5 internal calls>
    at java.base/java.lang.Thread.run(Thread.java:834)

Nov. 04, 2020 6:39:48 NACHM. org.junit.platform.launcher.core.TestExecutionListenerRegistry lambda$notifyEach$1
WARNING: TestExecutionListener [org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestExecutionListener] threw exception for method: executionFinished(TestIdentifier [uniqueId = '[engine:kotest]/[spec:com.example.core.common.JSONFileTests]/[test:Writing to and reading from file should result in the original value]', parentId = '[engine:kotest]/[spec:com.example.core.common.JSONFileTests]', displayName = 'Writing to and reading from file should result in the original value', legacyReportingName = 'Writing to and reading from file should result in the original value', source = FileSource [file = D:\dev\Minecraft Server Watcher\msw-server\JSONFileTests.kt, filePosition = FilePosition [line = 40, column = -1]], tags = [], type = TEST], TestExecutionResult [status = SUCCESSFUL, throwable = null])
java.lang.AssertionError <3 internal calls>
    at jdk.internal.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) <11 internal calls>
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1540) <17 internal calls>
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
    at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
    at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:84)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
    at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
    at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source) <5 internal calls>
    at java.base/java.lang.Thread.run(Thread.java:834)

Received a failure event for test with unknown id '2.27'. Registered test ids: '[2.1, :test, 2.25]'
java.lang.IllegalArgumentException: Received a failure event for test with unknown id '2.27'. Registered test ids: '[2.1, :test, 2.25]' <19 internal calls>
    at java.base/java.lang.Thread.run(Thread.java:834)
> Task :test FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':test'.
> Received a failure event for test with unknown id '2.27'. Registered test ids: '[2.1, :test, 2.25]'
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 5s
10 actionable tasks: 2 executed, 8 up-to-date

What might be relevant:

1. I'm using custom task configurations in my `build.gradle.kts`

The following section of my build.gradle.kts file covers all testing-related stuff, including a task to set up an environment for testing and a second one to cleanup afterwards:

tasks.withType<Test> {
    useJUnitPlatform()
}

allure {
    autoconfigure = false
    version = "2.13.6"
}

configure<PitestPluginExtension> {
    testPlugin.set("Kotest")
    targetClasses.set(listOf("com.example.*"))
}

val testDir1Path = "/src/test/resources/DirectoryTest"
val testDir2Path = "/src/test/resources/Create"
tasks.register("preTest") {
    doLast {
        mkdir(testDir1Path)
        file("$testDir1Path/json00.json").apply { createNewFile() }.writeText("{ \"s\": \"s\", \"i\": 0, \"b\": true}")
        file("$testDir1Path/json01.json").apply { createNewFile() }.writeText("{}")
        file("$testDir1Path/json02.json").apply { createNewFile() }.writeText("{}")
        file("$testDir1Path/json03.json").apply { createNewFile() }.writeText("{}")
        file("$testDir1Path/json04.json").apply { createNewFile() }.writeText("{}")
        file("$testDir1Path/json05.json").apply { createNewFile() }.writeText("{}")
        file("$testDir1Path/json06.json").apply { createNewFile() }.writeText("{}")
    }
}

tasks.register<Delete>("postTest") {
    for (task in tasks.withType<Test>()) {
        mustRunAfter(task)
    }
    delete(testDir1Path)
    delete(testDir2Path)
    delete("/src/test/resources/NonExistent") // might be created by :pitest
}

tasks.withType<Test>().configureEach {
    dependsOn("preTest")
}

2. The test suite that first caused the error

The error occurred while I was working in the following Kotest test class, more precisely it occurred after I tried to implement a Test Factory and include it:

class JSONFileTests : FunSpec({
    test("Loading a file into the model JSONModel01 should work") {
        shouldNotThrowAny {
            JSONFile("$exists/json00.json", JSONModel01.serializer())
        }.get() shouldBe JSONModel01("s", 0, true)
    }

    include("JSONModel01: ", echoTests(m1))
    include("JSONModel02: ", echoTests(m2))
    include("JSONModel03: ", echoTests(m3))
    include("JSONModel04: ", echoTests(m4))
    include("JSONModel05: ", echoTests(m5))
    include("JSONModel06: ", echoTests(m6))
}) {
    companion object {
        private val m1 = JSONFile("$exists/json01.json", JSONModel01.serializer()) to JSONModel01()
        private val m2 = JSONFile("$exists/json02.json", JSONModel02.serializer()) to JSONModel02()
        private val m3 = JSONFile("$exists/json03.json", JSONModel03.serializer()) to JSONModel03()
        private val m4 = JSONFile("$exists/json04.json", JSONModel04.serializer()) to JSONModel04()
        private val m5 = JSONFile("$exists/json05.json", JSONModel05.serializer()) to JSONModel05()
        private val m6 = JSONFile("$exists/json06.json", JSONModel06.serializer()) to JSONModel06()

        private fun cleanupAll() {
            m1.first.reload()
            m2.first.reload()
            m3.first.reload()
            m4.first.reload()
            m5.first.reload()
            m6.first.reload()
        }

        private fun <T : JSONModelMarker> echoTests(model: Pair<JSONFile<T>, T>) = funSpec {
            test("Writing to and reading from file should result in the original value") {
                model.first.apply { setAndCommit(model.second) }.get() shouldBe model.second
                model.first.reload()
            }
        }
    }
}

All the JSONModelXX classes are model data classes only intended for testing. JSONModelMarker is a marker interface implemented by all JSONModelXX classes.JSONFile<T> is the class I am trying to test.

3. IntelliJ notifications

I always get an "all tests successful" notification from IntelliJ, but when I look at the testing tool window, the tests are marked as "failed" because the gradle task :test failed. I don't know how this plays into the issue.

4. I sometimes also receive a different exception

If I only run the first test in the test suite via the IntelliJ gutter icon, I get a different exception: java.lang.ExceptionInInitializerError. I did some digging and found out that the Scanner that is used to read the contents of a JSON file in the JSONFile<T> class throws a NoSuchElementException because (for some reason) the file is empty only during execution of this test. That is pretty weird because the self-defined :preTest task in my build.gradle.kts actually writes some stuff into each file, and those contents are actually there when I look at the files after :preTest or :test ran. Here is the full console output:

java.lang.ExceptionInInitializerError
java.lang.ExceptionInInitializerError
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:315)
    at io.kotest.engine.launcher.ExecuteKt$execute$1.invokeSuspend(execute.kt:37)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlin.coroutines.ContinuationKt.startCoroutine(Continuation.kt:115)
    at io.kotest.engine.launcher.ExecuteKt.future(execute.kt:89)
    at io.kotest.engine.launcher.ExecuteKt.execute(execute.kt:34)
    at io.kotest.engine.launcher.MainKt.main(main.kt:12)
Caused by: java.util.NoSuchElementException
    at java.base/java.util.Scanner.throwFor(Scanner.java:937)

    at java.base/java.util.Scanner.next(Scanner.java:1478)
    at com.example.core.common.JSONFile$Companion.readFile(JSONFile.kt:14)
    at com.example.core.common.JSONFile$Companion.access$readFile(JSONFile.kt:11)
    at com.example.core.common.JSONFile.<init>(JSONFile.kt:27)
    at com.example.core.common.JSONFileTests.<clinit>(JSONFileTests.kt:24)
    ... 8 more


Process finished with exit code 0

[java.lang.ExceptionInInitializerError]

Now I don't know whether this is part of the issue or a separate issue, but just to be sure I wanted to include it here. The same issue occurs when I switch the test runner to IntelliJ IDEA in File->Settings->Build, Execution, Deployment->Build Tools->Gradle

Thats it! Now my question is: What is causing this error and how can I fix it?

Running with

kotlin/JVM 1.4.10
kotest-runner-junit5 4.3.1
kotest-assertions-core 4.3.1
kotest-property 4.3.1
kotest-extensions-allure 4.3.1
kotest-plugins-pitest 4.3.1
gradle 6.7
allure 2.13.6
allure gradle plugin 2.8.1
pitest gradle plugin 1.5.1
intellij IDEA Ultimate 2020.2
kotest IntelliJ plugin 1.1.20-IC-2020.2
openJDK 11.0.2

Have you tried changing test runner? https://www.jetbrains.com/help/img/idea/2020.2/gradle_test_runner.png — Михаил Нафталь, Nov 04 '20 at 20:33
I have not. Is it necessary? If anyhow possible, I'd like to continue testing via gradle. But I'll try it out — Raphael Tarita, Nov 04 '20 at 20:36
Please, check. Just to eliminate possibility of bug in Gradle plugin... — Михаил Нафталь, Nov 04 '20 at 20:37
Alright, I now tested it using the IntelliJ IDEA test runner and now I have the same issue as mentioned in #4. By now I'm pretty convinced that this is a separate issue, nonetheless I don't have a solution for it. — Raphael Tarita, Nov 04 '20 at 20:39
According to stacktrace, exceptions happens inside companion object of `JSONFile` class. Could you show its contents? — Михаил Нафталь, Nov 04 '20 at 20:51
As explained in #4, I already figured out what happens in this issue. But I don't know why the files are empty at the time the tests run — Raphael Tarita, Nov 04 '20 at 21:12

score 4 · Accepted Answer · answered Nov 07 '20 at 19:28

Seems like I found a solution.

The problem seems to be that Kotest (and later down the stream also JUnit) wants unique tests, which means unique test names. And normally, those problems would be marked by the Kotest IntelliJ plugin. But if you use a test factory, things are a bit more opaque. I was assuming that include(prefix, testFactory) adds the prefix to the test name, but this is not the case. which means that the factory method

private fun <T : JSONModelMarker> echoTests(model: Pair<JSONFile<T>, T>) = funSpec {
    test("Writing to and reading from file should result in the original value") {
        model.first.apply { setAndCommit(model.second) }.get() shouldBe model.second
        model.first.reload()
    }
}

actually creates a test with a fixed identifier every time it is called. Now, when I include this test factory multiple times via include(), I actually create duplicate tests which then crashes the test task. To solve this problem, I just implemented a functionality to alternate the test names, like this:

private fun <T : JSONModelMarker> echoTests(name: String, model: Pair<JSONFile<T>, T>) = funSpec {
    test("$name: Writing to and reading from file should result in the original value") {
        model.first.apply { setAndCommit(model.second) }.get() shouldBe model.second
        model.first.reload()
    }
}

Now, when I call the include() method, I omit the prefix (because it is irrelevant) and add an unique name (which was previously the prefix) to the factory method call:

include(echoTests("JSONModel01", m1))

After I did this, the error did not occur anymore.

Can you file this as a bug at kotest,io ? – sksamuel Dec 15 '20 at 01:37 — sksamuel, Dec 15 '20 at 01:37