Why does sbt compile fail with StackOverflowError?

Question

I am working on a Scala project that has been in existence for a few years but is new to me. My task is to upgrade it from Scala 2.9.3 to 2.11.7, along with its dependencies. I have gotten past the errors and warnings, but I cannot get the project to compile successfully in SBT. I always get a StackOverflowError in pretty much the same place. The stacktrace looks like this, but details vary with the Xss setting (currently 4M, but have tried as high as 24M):

java.lang.StackOverflowError
at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:698)
at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5395)
at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5422)
at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5369)
at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5373)
at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5471)
at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5479)
at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:644)
at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:698)
at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5395)
at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5422)

SBT_OPTS looks like this:

-Xmx2G -Xss4M -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled

I can 'make' the project successfully in Intellij, and others can pull my changes from GitHub and compile the project in sbt, so the issue seems to be local to my machine (a recent quad-core Macbook Pro with 16GB RAM). Other Scala/sbt projects compile successfully for me on this machine.

Here are other relevant details:

Scala version: 2.11.7
Java version: java version "1.8.0_66" (build 1.8.0_66-b17)
sbt version: 0.13.7 (have also tried 0.13.9)

I have completely rebuilt the ivy2 cache and cleared the lib_managed directory. The version of the scala-compiler.jar is the same as is used on at least one machine that can 'sbt compile' the code successfully. I did a clean reinstall of sbt (via brew remove sbt, manual removal of ~/.sbt directory, then brew install sbt).

I have not tried to isolate the line of source code being compiled when the error occurs. I have assumed it would be more productive to look for a configuration issue or dependency conflict somewhere.

Any suggestions for further troubleshooting will be appreciated.

[Added...] It may be helpful to add that, as an experiment, I downloaded the Scala language source code from https://github.com/scala/scala and got the following very similar error trying to sbt compile it:

java.lang.StackOverflowError
at scala.tools.nsc.transform.ExplicitOuter$OuterPathTransformer.outerValue(ExplicitOuter.scala:229)
at scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:441)
at scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:352)
at scala.reflect.internal.Trees$class.itransform(Trees.scala:1345)
at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2555)
at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:44)
at scala.tools.nsc.transform.ExplicitOuter$OuterPathTransformer.scala$reflect$internal$Trees$UnderConstructionTransformer$$super$transform(ExplicitOuter.scala:219)
at scala.reflect.internal.Trees$UnderConstructionTransformer$class.transform(Trees.scala:1693)
at scala.tools.nsc.transform.ExplicitOuter$OuterPathTransformer.transform(ExplicitOuter.scala:291)
at scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:459)
at scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:352)
at scala.reflect.internal.Trees$class.itransform(Trees.scala:1347)
at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2555)

Here is something interesting. From this post I found out about launching sbt with a -d flag for debugging info. Got the following output:

Kevins-MacBook-Pro:scala kdoherty$ sbt -d
[process_args] java_version = '1.8.0_66'
# Executing command line:
java
-Xmx2G
-Xss4M
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-Xmx384m
-Xss512k
-XX:+UseCompressedOops
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
-jar
/usr/local/Cellar/sbt/0.13.9/libexec/sbt-launch.jar

So somewhere my SBT_OPTS settings are being overridden (by defaults, I guess). Now I need to find where those defaults are coming from.

When you say you have the same version of scala-compiler.jar as another computer that can compile this, do you mean that `scalac -version` returns the same thing on both computers, or that both computers have 2.11.7 defined in the .sbt build file? — childofsoong, Dec 16 '15 at 17:59
@KevinDoherty Do you happen to know the version of java installed on this other computer, and what all of the JVM settings for it are? — childofsoong, Dec 16 '15 at 18:03
@soong My other Mac (13-inch MBP), which will compile the project successfully, is using Java 1.8.0-b132. No special JVM settings are used, just the defaults. — Kevin Doherty, Dec 16 '15 at 18:14
they don't look like they are getting overwritten, they are getting duplicated! (I see both memory properties getting listed twice but with different values) — Andrew Norman, Apr 18 '18 at 21:10

score 47 · Answer 1 · answered Dec 16 '15 at 20:53

47

I figured it out. Once I knew that the -d flag would tell me what settings SBT was actually using, I saw that the values in my SBT_OPTS environment variable were being clobbered by other, lower settings. Where were those coming from? From my JAVA_OPTS env variable! I should have noticed them sooner, but now I know I can keep those Java options as they are and override them by adding the SBT-specific settings to my /usr/local/etc/sbtopts file, using the somewhat awkward format of

-J-Xmx2G
-J-Xss2M

Using the values shown I was able to run sbt compile successfully on my project.

I hope someone finds this useful.

answered Dec 16 '15 at 20:53

Kevin Doherty

987
1
7
7

If I run this: sbt -Pyarn -pHadoop 2.7 assembly -J-Xss2M I get: Java HotSpot(TM) Client VM warning: ignoring option MaxPermSize=256m; support wa s removed in 8.0 - Any ideas? – cs0815 Feb 04 '17 at 16:21
Upgrade sbt and the warning goes away. – Jacek Laskowski Feb 20 '18 at 12:46
MaxPermSize was replaced in java 8 with -XX:MaxMetaspaceSize – Andrew Norman Mar 27 '18 at 00:12
it would be nice if there was a solution that works inside the sbt project and therefore could be run anywhere on anything that is trying to do the build – Andrew Norman Apr 18 '18 at 21:13
the sbt -d param doesn't work for me I get a "The `-` command is deprecated in favor of `onFailure` and will be removed in 0.14.0". That said, "sbt show scalacOptions" is showing the correct memory settings for me. My problem is that the compiler is still somehow ignoring / not honoring the values. – Andrew Norman Apr 19 '18 at 20:55

score 10 · Answer 2 · answered Jul 08 '19 at 03:43

10

Add to the bottom of /usr/local/etc/sbtopts

-J-Xmx4G 
-J-Xss4M

All set.

answered Jul 08 '19 at 03:43

Ajit Surendran

709
7
4

score 7 · Answer 3 · answered Sep 04 '19 at 09:20

7

I just added -Xss in my Intellij sbt properties and the issue is resolved.

Intellij SBT properties

answered Sep 04 '19 at 09:20

Pradyumna_Kumar

71
1
1

score 5 · Answer 4 · answered Oct 27 '19 at 22:47

Relevant parts from the output of sbt -h:

  # jvm options and output control
  JAVA_OPTS          environment variable, if unset uses ""
  .jvmopts           if this file exists in the current directory, its contents
                     are appended to JAVA_OPTS
  SBT_OPTS           environment variable, if unset uses ""
  .sbtopts           if this file exists in the current directory, its contents
                     are prepended to the runner args

Thus, in my case I solved the problem by creating a file .sbtopts with content

-J-Xmx4G 
-J-Xss4M

in the project directory.

Note: Running sbt -d shows what settings have been used, for instance:

$ sbt -d
[addSbt] arg = '-debug'
[process_args] java_version = '8'
# Executing command line:
java
-Xms1024m
-XX:ReservedCodeCacheSize=128m
-XX:MaxMetaspaceSize=256m
-Xmx2G
-Xss2M
-jar
/path/to/sbt-launch.jar
-debug

Robbie Milejczak · Answer 5 · 2019-07-08T12:54:19.010

4

I was unable to get this to work via the provided answers. build.sbt settings and the sbtopts file both failed to solve this error for me. What I had to do was run sbt with the -mem flag, for example:

sbt -mem 2048 compile

and now my projects built. If using IntelliJ you can also go to

Preferences > Build, Execution, Deployment > Build Tools > sbt

and set Maximum heap size, MB to whatever target you need.

edited Jul 08 '19 at 12:54

answered Jul 03 '19 at 15:45

Robbie Milejczak

5,664
3
32
65

score 1 · Answer 6 · answered Apr 19 '18 at 22:58

I discovered the SBT_OPT setting in the bin/sbt file of my sbt install was affecting the memory values set in my projects build.sbt

updating the existing -Xss value in this file from 1M to 8M raised the memory size of the Scalac stack to a point I stopped getting StackOverflow exceptions in the sbt-invoked compiler. This seemed odd because the sbt documented approach to setting stack size in the compiler is to do this is with the -J-Xss setting.

Sbt doesn't seem to actually enable you to set the compiler's stack memory. While a build.sbt accepts the following configuration as a valid setting, it doesn't seem to apply the value in the compiler:

scalacOptions in ThisBuild ++= Seq(-J-Xss8M)

I suspect that is a bug or non-implemented functionality

GJ. · Answer 7 · 2020-08-10T15:56:33.287

1

I too run into this problem recently and I discovered a working solution for it with sbt 1.3.13.

create a .sbtopts file under project root
add -J-Xss100M (or any thread stack size you think suitable) to the .sbtopts file

edited Aug 10 '20 at 15:56

answered Aug 10 '20 at 15:40

GJ.

870
5
9
26

Pavan_Obj · Answer 8 · 2021-01-28T16:56:17.080

0

There are multiple Correct Answers already. But what worked for me is below,

// Created and Added a File: .jvmopts in the Project Root Folder with below Parameters.
-Xms3022m
-Xmx4048m
-Xss124m
-XX:MaxPermSize=4048m
-XX:MaxMetaspaceSize=512m
-XX:+CMSClassUnloadingEnabled
-XX:ReservedCodeCacheSize=128m

edited Jan 28 '21 at 16:56

answered Jul 07 '20 at 19:10

Pavan_Obj

1,071
1
12
26

1

Why do you have MaxMetaspaceSize twice? – dustin Jan 26 '21 at 03:42

Why does sbt compile fail with StackOverflowError?

8 Answers8

Linked