sbt compile takes very long time to complete

Question

I’m now running a comparatively small project with The Akka, Play Framework, Scala and build tool is SBT. However, I have built a large project than this one and it has built pretty quickly. But now after some time I’m trying to build my project and it’s taking long time to compile (Total time: 904 s).

> ~/nlp-search/code(branch:feature/demo-app-api*) » sbt compile         
> renienj@Rajani [info] Loading project definition from
> /Users/renienj/nlp-search/code/project [info] Set current project to
> nlp-search (in build file:/Users/renienj/nlp-search/code/) [info]
> Compiling 115 Scala sources and 1 Java source to
> /Users/renienj/nlp-search/code/target/scala-2.11/classes... [warn]
> /Users/renienj/nlp-search/code/app/api/parsedquery/ParsedQuery.scala:14:
> imported `format' is permanently hidden by definition of value format
> in object ParsedQuery [warn]   import Sentence.format [warn]          
> ^ [warn]
> /Users/renienj/nlp-search/code/app/api/parsedquery/Sentence.scala:14:
> imported `format' is permanently hidden by definition of value format
> in object Sentence [warn]   import Phrase.format [warn]               
> ^ [warn]
> /Users/renienj/nlp-search/code/app/api/parsedquery/Sentence.scala:15:
> imported `format' is permanently hidden by definition of value format
> in object Sentence [warn]   import  Filter.format [warn]              
> ^ [warn]
> /Users/renienj/nlp-search/code/app/api/parsedquery/Sentence.scala:16:
> imported `format' is permanently hidden by definition of value format
> in object Sentence [warn]   import Order.format [warn]               
> ^ [warn]
> /Users/renienj/nlp-search/code/app/backend/intelligenceServices/nlp/executor/StreamExecutor.scala:23:
> abstract type pattern OT is unchecked since it is eliminated by
> erasure [warn]     case x: OT => [warn]             ^ [warn]
> /Users/renienj/nlp-search/code/app/backend/intelligenceServices/merchRules/io/DAO.scala:27:
> method newTermName in trait Names is deprecated: Use TermName instead
> [warn]       String.valueOf(x.map(x => x.toChar)).unpickle[MerchRule]
> [warn]                                                    ^ [warn]
> /Users/renienj/nlp-search/code/app/backend/solrAccess/cache/SolrCacheProvider.scala:24:
> method newTermName in trait Names is deprecated: Use TermName instead
> [warn]       x.pickle.value [warn]         ^ [warn]
> /Users/renienj/nlp-search/code/app/backend/solrAccess/cache/SolrCacheProvider.scala:30:
> method newTermName in trait Names is deprecated: Use TermName instead
> [warn]       BinaryPickle(x).unpickle[DocumentList] [warn]            
> ^ [warn] there were 9 feature warnings; re-run with -feature for
> details [warn] 9 warnings found [success] Total time: 904 s, completed
> Apr 9, 2016 5:07:09 PM

My build.sbt :

name := """nlp-search"""

version := """3.0-RC1"""

lazy val root = (project in file(".")).enablePlugins(PlayScala)

scalaVersion := "2.11.7"

resolvers += Resolver.mavenLocal

val akkaVersion = "2.3.12"

libraryDependencies ++= Seq(
  "com.typesafe.akka"     %% "akka-actor"       % akkaVersion,
  "com.typesafe.akka"     %% "akka-contrib"     % akkaVersion,
  "com.typesafe.akka"     %% "akka-remote"      % akkaVersion,
  "com.typesafe.akka"     %% "akka-cluster"     % akkaVersion,
  "com.typesafe.akka"     %% "akka-testkit"     % akkaVersion,
  "org.apache.solr"       % "solr-solrj"        % "4.10.0",
  "org.apache.thrift"     % "libthrift"         % "0.9.2",
  "org.json"              % "json"              % "20080701",
  "org.skife.com.typesafe.config" % "typesafe-config" % "0.3.0",
  "net.debasishg"         %% "redisclient"      % "2.13",
  "org.scala-lang.modules" %% "scala-pickling"  % "0.10.0",
  "edu.stanford.nlp"      % "stanford-parser"   % "3.5.2",
  "edu.stanford.nlp" % "stanford-corenlp" % "3.5.2",
  "org.scalatest"         % "scalatest_2.11"    % "2.2.4" % "test",
  "com.kohls.search.qp"   % "api"               % "1.0.0",
  "com.typesafe.akka"     %% "akka-stream-experimental" % "1.0"
)

libraryDependencies += specs2 % Test

resolvers += "scalaz-bintray" at "https://dl.bintray.com/scalaz/releases"

publishArtifact in (Compile, packageDoc) := false

publishArtifact in packageDoc := false

sources in (Compile,doc) := Seq.empty

I’m struggling from morning. I really can’t understand why it takes so long.

Thank you very much in advance.

Does it contain a lot of macro implicits generating JSON readers and writers on the fly? Was a killer to me. I cached the implicits and shortened my build times from 5+ minutes to about 30 seconds. — Jean-Philippe Pellet, Apr 09 '16 at 13:00
Yes, I have like 8 implicits to generate complex JSON. Can you please show me a sample to try it out. Did you used shapeless lib? Thanks for your comment. — Renien, Apr 09 '16 at 13:50
I'm using upickle and bsonpickle. I precompute and cache all my implicits like so: `object JsonSerialization { implicit lazy val userReader = { val userReader = (); macroR[User] }; … }` — Jean-Philippe Pellet, Apr 09 '16 at 19:06
You may also want to have a look at `cachedImplicit` in shapeless. — Jean-Philippe Pellet, Apr 09 '16 at 19:07
Thanks a lot @Jean-PhilippePellet. Why don't you move the comment to answer so the question will be complete. — Renien, Apr 11 '16 at 08:13

score 3 · Answer 1 · edited May 23 '17 at 12:13

Not really an answer, but maybe a hint… If you have a lot of implicits that are generated by macro applications (typically, JSON readers and writers for your own case classes and the like), compilation can be slowed down considerably. You can try caching the implicits yourself with something like this (example with upickle):

object JsonSerialization {
  implicit lazy val userReader = { val userReader = (); macroR[User] }
  implicit lazy val userWriter = { val userWriter = (); macroR[Writer] }
}

You may also want to have a look at shapeless's cachedImplicit, also mentioned in this question.

sbt compile takes very long time to complete

1 Answers1