1

I am currently using ScalaTest 3.0.1 for my Scala 2.11.8 project, with sbt 0.13.18 as the build tool. The IDE is Intellij.

The project is a Spark project, and I'm using a temp directory at root/temp/ for checkpoints and warehousing. When I run the tests multiple times the checkpoints keep getting added to, eventually reaching a very large size. I want to keep the checkpoints around after the test runs so they can be used for verification, but I would like to delete them before the next run.

How can you accomplish this using sbt and ScalaTest in Intellij?

I've been looking into setting up an sbt clean with my temp directory in build.sbt but I can't seem to get it to delete the directory, and reading the documentary hints it will only delete the files sbt has created, so I abandoned that idea.

I'm currently looking into setting up my ScalaTest set up to have a BeforeAll event which will delete the directory, but I'm not sure this is the correct approach and I've been having issues getting it to work.

WarSame
  • 870
  • 1
  • 10
  • 24
  • "The IDE is Intellij." What about it? – pedrofurla Jul 09 '20 at 15:35
  • Including that in case it's relevant, because I've noticed that Intellij sometimes treats sbt weirdly. I also just realized that I've left the versions off, so I'll go add those in. – WarSame Jul 09 '20 at 16:03

1 Answers1

2

Based on https://stackoverflow.com/a/48659771/5205022 clean can include temp directory

cleanFiles += baseDirectory.value / "temp"

Another option is to create a custom task that cleans temp, for example using better-files

lazy val deleteTestTemp = taskKey[Unit]("Delete test temp directory")
deleteTestTemp := {
  import better.files._
  val temp = (baseDirectory.value / "temp").toScala
  if (temp.exists) temp.delete()
}

where project/plugins.sbt contains

libraryDependencies += "com.github.pathikrit" %% "better-files" % "3.9.1"

we could clean and then test like so

deleteTestTemp;test
Mario Galic
  • 47,285
  • 6
  • 56
  • 98
  • The first approach is the exact same code as I had before but is not working. When I `inspect clean-files` I do see the directory listed, but I don't see any details when I use the `last` command. It simply runs quickly and doesn't delete any files. I am trying to get the second approach working but can't seem to import better.files._ even though I have the module. – WarSame Jul 09 '20 at 17:52
  • @WarSame Regarding the first approach, if you execute `show cleanFiles` does it list the `temp` directory? Regarding the second approach, make sure to add the dependency under `project/plugins.sbt` or `project/build.sbt` as opposed to `builld.sbt` of the root project, because it need to be a dependency of the meta-project as opposed to project proper as per https://www.scala-sbt.org/1.x/docs/Organizing-Build.html#sbt+is+recursive – Mario Galic Jul 09 '20 at 17:59
  • If I execute `show clean-files` or `show cleanFiles` I do see the `temp` directory listed in both though it still isn't getting deleted. If I put the `libraryDependency` you mentioned in my `project/plugins.sbt` Intellij gives a hint that `Expression type Def.Setting[Seq[ModuleID]] must conform to DslEntry in sbt file` and cannot resolve the dependency. I tried `addSbtPlugin` which had the same issue. I can't seem to get it to actually find this dependency. – WarSame Jul 09 '20 at 18:47