Sequential test setup/cleanup in sbt

Question

I have an SBT project with multiple subprojects aggregated within a root project. I want to make sure that test suites are ran sequentially, so I have set Global / concurrentRestrictions += Tags.limit(Tags.Test, 1).

However, I am also using Tests.Setup()/Cleanup() to initialise memory-heavy resources for each subproject's test suite. The problem I'm facing is that concurrentRestrictions has no effect on test setup and cleanup: all of my test setup and cleanup logic is being ran in parallel, when I would like it to run sequentially before and after each test suite.

Here is a sample build.sbt:

import sbt._
import scala.sys.process._

lazy val testSettings = Seq(
    testOptions ++= Seq(
      Tests.Setup(() => {
        println(s"setup for ${name.value}")
      }),
      Tests.Cleanup(() => {
        println(s"cleanup for ${name.value}")
      })
    )
)

Global / libraryDependencies ++= Seq(
  "org.scalatest" %% "scalatest" % "3.1.0" % "test"
)

def module(path: String) = {
  Project(path, file(path))
    .settings(testSettings)
}

lazy val module1 = module("module1")

lazy val module2 = module("module2")

lazy val module3 = module("module3")

lazy val root = (project in file("."))
  .aggregate(module1, module2, module3)

Global / concurrentRestrictions += Tags.limit(Tags.Test, 1)

and a dummy test I'm using for each subproject:

import org.scalatest.matchers.should.Matchers
import org.scalatest._

class DummyTest extends FlatSpec with Matchers {
  "this test" should "run a test for module 1" in {}
}

Running sbt test gives the following output, modulo the vagaries of concurrency:

$ sbt test
[info] Loading settings for project global-plugins from plugins.sbt ...
[info] Loading global plugins from /home/jejost/.sbt/1.0/plugins
[info] Loading project definition from /home/jejost/dev/tmp/sbt-yunowork/project
[info] Loading settings for project root from build.sbt ...
[...]
setup for module1
setup for module3
setup for module2
[info] DummyTest:
[info] this test
[info] - should run a test for module 1
[info] DummyTest:
[info] this test
[info] - should run a test for module 3
[info] DummyTest:
[info] this test
[info] - should run a test for module 2
cleanup for module1
cleanup for module3
cleanup for module2
[...]

when the behaviour I want is this:

$ sbt test
[info] Loading settings for project global-plugins from plugins.sbt ...
[info] Loading global plugins from /home/jejost/.sbt/1.0/plugins
[info] Loading project definition from /home/jejost/dev/tmp/sbt-yunowork/project
[info] Loading settings for project root from build.sbt ...
[...]
setup for module1
[info] DummyTest:
[info] this test
[info] - should run a test for module 1
cleanup for module1
setup for module3
[info] DummyTest:
[info] this test
[info] - should run a test for module 3
cleanup for module3
setup for module2
[info] DummyTest:
[info] this test
[info] - should run a test for module 2
cleanup for module2

[...]

(Ordering of module1, module3, module2 doesn't matter to me as long as setup/teardown is sequential before/after each subproject)

Is it possible to ensure sequential test setup and cleanup like this, ideally without defining my own custom task?

I am using SBT 1.3.4.

EDIT: this has been flagged as a duplicate of the following question: How to run subprojects tests (including setup methods) sequentially when testing This is indeed the same problem, however, the accepted answer does not work and does not change anything to the task run order! I tried to reproduce the other question's setup to see if it's a regression in SBT, unfortunately the OP's SBT version is too old to build successfully on my machine.

Does this answer your question? [How to run subprojects tests (including setup methods) sequentially when testing](https://stackoverflow.com/questions/18895063/how-to-run-subprojects-tests-including-setup-methods-sequentially-when-testing) — Shankar Shastri, Jun 02 '20 at 18:48
No it doesn't: I don't know if it's due to a behaviour change/regression in SBT, but because test setup/cleanup runs in its own separate task, adding `parallelExecution in Global := false` ensures that only one of the subproject's test setup task runs at the same time, *but it doesn't prevent the other subproject setup tasks to run before that project's tests and cleanup have finished*. — jjst, Jun 03 '20 at 07:29
In practice, I am seeing no tasks being ran in parallel with `parallelExecution in Global := false`, but no change in execution order, and no change whatsoever when adding `parallelExecution in ThisBuild := false` — jjst, Jun 03 '20 at 07:29
I am afraid that you cannot achieve this, as elaborated in [Forking tests](https://www.scala-sbt.org/1.x/docs/Testing.html#Forking+tests): _`Setup` and `Cleanup` actions cannot be provided with the actual test class loader when a group is forked._ I know that you are not using groups, but if you cannot do it with groups probably you cannot do it at all. — Tomer Shetah, Mar 11 '21 at 22:24

Mario Galic · Answer 1 · 2021-03-12T11:58:37.187

Commands, unlike tasks, are by default executed sequentially so consider simply dropping down to command level

addCommandAlias("sequentialtest", ";module1/test;module2/test;module3/test;")

which gives something like

sbt:hello-world-scala> sequentialtest
setup for module1
[info] DummyTest:
[info] this test
[info] - should run a test for module 1
cleanup for module1
[info] Run completed in 164 milliseconds.
...
setup for module2
[info] DummyTest:
[info] this test
[info] - should run a test for module 2
cleanup for module2
[info] Run completed in 148 milliseconds.
...
setup for module3
[info] DummyTest:
[info] this test
[info] - should run a test for module 3
cleanup for module3
[info] Run completed in 154 milliseconds.
...

Addressing the comment

commands += Command.command("sequentialtest") { state =>
  Project
   .extract(state)
   .structure
   .allProjects
   .iterator
   .map(_.id)
   .filterNot(_ == "root")
   .map(id => s"$id/test;")
   .mkString :: state
}

That's a slight improvement, I guess - is there a way to make this work without tying it specifically to subproject names, or how many of them exist? In other words, is there any way to make this work even when I add "module4" or remove "module2" without having to change the command alias? — jjst, Mar 12 '21 at 11:27
Okay, that edit is pretty impressive :) thanks a lot, going to try this out. — jjst, Mar 17 '21 at 14:14

Sequential test setup/cleanup in sbt

1 Answers1