sbt: dynamic aggregation of subproject

Question

I would like to invent a system to dynamically discover subprojects and aggregate them into my project automatically. Or at least configure this somehow.

To do this, I'm planning to have either a "modules" folder or an optional configuration file containing paths to the modules.

In any case I'd need to loop through subfolders (or loop through a list of paths from a configuration file), and aggregate each subproject. I don't know how to do that.

Currently I'm building in the Play framework with the build.sbt file. I would need to add the loop like this:

name := "mysite"
version := "1.0"
scalaVersion := "2.11.1"
lazy val root = (project in file(".")).enablePlugins(PlayJava)
//pseudocode:
foreach( folder in the 'modules' folder) { 
  lazy val module = (project in file(folder)).enablePlugins(PlayJava)
  root = root.dependsOn(module).aggregate(module)
}

Is there a way to do this?

EDIT 3: The code here is almost working:

object MyBuild extends Build {
  name := "mysite"
  version := "1.0"
  scalaVersion := "2.11.6"

  var m = new File("modules")
  var list = Seq[ProjectReference]()
  var deps = Seq[ClasspathDependency]()
  if (m.exists) {
    val subs = m.listFiles.filter ( _.isDirectory ).foreach { folder =>
      var modulePath = new File("modules", folder.getName)
      println("Found module " + modulePath)
      lazy val module:ProjectRef = ProjectRef(modulePath,folder.getName)
      lazy val dep:ClasspathDependency = ClasspathDependency(module, None)
      list = list :+ module
      deps = deps :+ dep
    }
  }

  lazy val root = Project(id = "mysite", base = file(".")).enablePlugins(PlayJava).aggregate(list:_*).dependsOn(deps:_*)
}

Edit 4:

See Dale Wijnand's solution below.

About the error: RuntimeException: No project 'myModule' in 'file:/Users/me/mysite/modules/myModule'. I fixed this using the solution from https://stackoverflow.com/a/28820578

What is the output of `println("Found module " + folder.getName)`? — Peanut, Jun 24 '15 at 19:18
it's "Found module foo" (my module folder is named "foo"). It seems that r.depends(module) fails. It may not be a Project object. For some reason. — Emmanuel, Jun 24 '15 at 19:30

score 3 · Accepted Answer · edited May 23 '17 at 12:06

3

Here:

project/Build.scala

import sbt._
import sbt.Keys._

import play.sbt._
import play.sbt.Play.autoImport._

object Build extends Build {
  val commonSettings: Seq[Setting[_]] = Seq(
    scalaVersion := "2.11.7"
  )

  lazy val modules = (file("modules") * DirectoryFilter).get.map { dir =>
    Project(dir.getName, dir).enablePlugins(PlayJava).settings(commonSettings: _*)
  }

  lazy val root = (project in file("."))
    .enablePlugins(PlayJava)
    .settings(
      name := "mysite",
      version := "1.0"
    )
    .settings(commonSettings: _*)
    .dependsOn(modules map (m => m: ClasspathDependency): _*)
    .aggregate(modules map (m => m: ProjectReference): _*)

  override lazy val projects = root +: modules
}

Note, make sure that the module directories don't also contain build.sbt files defining them as projects as that will cause confusing RuntimeException: No project 'x' in 'file:/x' type exception, see Can't use sbt 0.13.7 with Play subprojects

edited May 23 '17 at 12:06

Community

1
1

answered Jun 25 '15 at 21:20

Dale Wijnand

6,054
5
28
55

That's the most complete solution so far. But I'm still getting the same error: RuntimeException: No project 'myModule' in 'file:/Users/me/mysite/modules/myModule'. Any idea why that would be? – Emmanuel Jun 26 '15 at 15:03
Note: the subproject itself can be compiled fine on its own. – Emmanuel Jun 26 '15 at 15:15
I've got it! The solution is here: http://stackoverflow.com/a/28820578 Basically the project root definition is duplicate in the main build and sub-project builds. The solution is to remove the root project definition in build.sbt for sub-projects. //lazy val root = ... – Emmanuel Jun 26 '15 at 15:25
Indeed I was typing about that and asking you if you could check. I'll add it as a NB to the answer. Would that be a complete solution for you then? – Dale Wijnand Jun 26 '15 at 15:27
A related question, if you're interested: http://stackoverflow.com/questions/31120440/play-framework-add-routes-from-sub-projects-dynamically – Emmanuel Jun 29 '15 at 16:11

Peanut · Answer 2 · 2015-06-26T06:23:11.243

0

You can try something like this:

import sbt._
import sbt.Keys._

import play.sbt._
import play.sbt.Play.autoImport._

object Build extends Build {

  lazy val modules = (file("modules") * DirectoryFilter).get.map { dir =>
    Project(dir.getName, dir).enablePlugins(PlayJava)
  }

  lazy val root = (project in file("."))
    .enablePlugins(PlayJava)
    .settings(
      name := "mysite",
      version := "1.0"
    )
    .dependsOn(modules map (m => m: ClasspathDependency): _*)
    .aggregate(modules map (m => m: ProjectReference): _*)

  override lazy val projects = root +: modules
}

Note: Inspired by Dale's answer, but I had to remove the "commonSettings", otherwise it would not work for me.

edited Jun 26 '15 at 06:23

answered Jun 24 '15 at 06:55

Peanut

3,753
3
31
45

I tried something similar. But root=root.dependsOn() doesn't work; that val is immutable. I get a "reassignment to val" error. I'm a newbie to Scala so please hold my hand with this. – Emmanuel Jun 24 '15 at 18:18
The solution is to make the root a "var" instead of "val". Now I've got something almost working but I get a different issue. See my updated code above. – Emmanuel Jun 24 '15 at 18:59
I'm sorry; I'm lost in this code. I don't know what the "t" and the "r" are. – Emmanuel Jun 24 '15 at 20:02
I have a solution that works on Windows but not on Mac. It's all about folders at this point. I'll have it figured out soon. – Emmanuel Jun 25 '15 at 13:27
OK I've updated my solution (see above) and I've narrowed down the problem to file() returning the wrong folder. It returns the root folder of the project and NOT the module folder. getCanonicalPath returns /Users/me/mysite/ instead of /Users/me/mysite/modules/myModule. This is very weird. – Emmanuel Jun 25 '15 at 13:59
Is the code you posted above int he build.sbt-file? – Peanut Jun 25 '15 at 14:21
I'm getting: `java.lang.RuntimeException: No project 'test1' in 'file:/home/ubuntu/workspace/'.` which suggests, that there is no project definition here for project "test1". It's probably looking for a lazy val which does not exist. – Peanut Jun 25 '15 at 14:42
No it's the file project/Build.scala. I'm getting that error too, but printing the file's canonical path tells me that the sub-modules' Project uses the wrong path somehow. – Emmanuel Jun 25 '15 at 14:52
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/81542/discussion-between-emmanuel-and-peanut). – Emmanuel Jun 25 '15 at 14:52
I came up with another possible solution above. This time the folders are 100% correct. But it still refuses to load the subproject. I'm really losing patience. – Emmanuel Jun 25 '15 at 18:19
@Peanut You're missing a .get after filter to get a `Seq[File]` and then you can drop the file in `file(folder)` as it's already a file. – Dale Wijnand Jun 25 '15 at 20:29
Btw commonSettings probably didn't work for you as its using the improved 0.13.8 syntax. – Dale Wijnand Jun 27 '15 at 11:28
Yes. I use 0.13.5 and 0.13.7 depending on the stuff I work with. – Peanut Jun 27 '15 at 11:29

sbt: dynamic aggregation of subproject

2 Answers2

Linked