Why does a large array constructor call break the Scala compiler?

Question

I'm doing some image processing for a job and was scripting things with Im4Java. In order to write some unit cases I decided to just store a local image as a byte array in the code. I grabbed a simple test image

And then converted it to the byte array like so:

import java.nio.file.{Paths, Files}
import java.nio.charset.StandardCharsets
val stuff =scala.io.Source.fromFile(fileName)(scala.io.Codec.ISO8859).map(_.toByte).toArray
Files.write(Paths.get("tmp.txt"), stuff.mkString(",").getBytes(StandardCharsets.UTF_8))

And then put those bytes into an Array[Byte] apply constructor:

class testCase() {
    val smallFile = Array[Byte](-1,-40,-1, …) // 9816 arguments to the constructor passed here
}

When I tried to compile this (see the full scala file in this gist) I was surprised to see it break the compiler's typer! I get the following stacktrace (I've truncated it a bit):

[error] uncaught exception during compilation: java.lang.StackOverflowError
java.lang.StackOverflowError
at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1241)
at scala.reflect.internal.Symbols$Symbol.baseTypeSeqLength$1(Symbols.scala:1628)
at scala.reflect.internal.Symbols$Symbol.isLess(Symbols.scala:1631)
at scala.reflect.internal.Types$Type.baseTypeIndex(Types.scala:992)
at scala.reflect.internal.Types$CompoundType.baseType(Types.scala:1655)
at scala.reflect.internal.Types$ClassTypeRef$class.baseType(Types.scala:2186)
at scala.reflect.internal.Types$TypeRef$$anon$6.baseType(Types.scala:2544)
at scala.reflect.internal.Types$class.firstTry$1(Types.scala:6064)
at scala.reflect.internal.Types$class.isSubType2(Types.scala:6228)
at scala.reflect.internal.Types$class.isSubType(Types.scala:5837)
at scala.reflect.internal.SymbolTable.isSubType(SymbolTable.scala:13)
at scala.reflect.internal.Types$class.fourthTry$1(Types.scala:6223)
at scala.reflect.internal.Types$class.thirdTryRef$1(Types.scala:6123)
at scala.reflect.internal.Types$class.thirdTry$1(Types.scala:6145)
at scala.reflect.internal.Types$class.secondTry$1(Types.scala:6109)
at scala.reflect.internal.Types$class.firstTry$1(Types.scala:6070)
at scala.reflect.internal.Types$class.isSubType2(Types.scala:6228)
at scala.reflect.internal.Types$class.isSubType(Types.scala:5837)
at scala.reflect.internal.SymbolTable.isSubType(SymbolTable.scala:13)
at scala.reflect.internal.Types$Type.$less$colon$less(Types.scala:872)
at scala.tools.nsc.typechecker.Typers$Typer.adapt(Typers.scala:1091)
at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5660)
at scala.tools.nsc.typechecker.Typers$Typer.typedArg(Typers.scala:3042)
at scala.tools.nsc.typechecker.Typers$Typer.loop$1(Typers.scala:3069)
at scala.tools.nsc.typechecker.Typers$Typer.loop$1(Typers.scala:3071)
at scala.tools.nsc.typechecker.Typers$Typer.loop$1(Typers.scala:3071)
at scala.tools.nsc.typechecker.Typers$Typer.loop$1(Typers.scala:3071)
at scala.tools.nsc.typechecker.Typers$Typer.loop$1(Typers.scala:3071)
at scala.tools.nsc.typechecker.Typers$Typer.loop$1(Typers.scala:3071)
(etc as this is where the stackoverflow happens)

I'm using the Scala 2.10.5 compiler and java jdk1.7.0_79

I'm probably going to just use a smaller image or something to fix my issue, but I would like to know why the compiler does this and if it can be fixed?

Normally, the image itself can't be the problem, since the compiler won't know about it at compile-time. — Clashsoft, Jul 22 '15 at 19:10
I doubt it has something to do with the actual image since error is thrown during compile time. Can you provide full version of code (it's unclear how `fileName`, `Files` and `Paths` are defined)? — om-nom-nom, Jul 22 '15 at 19:11
@Clashsoft there is no actual problem with the image. I just provided the code so that someone could reproduce the byte array I was using in the gist file if they wanted to in the same way. — EdgeCaseBerg, Jul 22 '15 at 19:12
Maybe you hit the "64K bytecode per method" limit the JVM has? Or, well, you're passing a bit over 9000 arguments to that constructor; some sources seem to say there's a limit at 255. — Lynn, Jul 22 '15 at 19:12
yes. I resized the image to 25x25 with imagemagick and I could use that byte array fine. 50x50 or 100x100 caused the crashes still though. @Mauris do you have a citation for that limit? It would be good to have for a possible answer on how big an array can be for the compiler itself I think — EdgeCaseBerg, Jul 22 '15 at 19:14
The problem might be that the compiler has to check all these arguments for being compatible with `Byte`. I don't know exactly how the Scala Compiler does this, but the stack trace hints at a recursive method (which is, if you ask me, a very bad compiler design) — Clashsoft, Jul 22 '15 at 19:15
@Clashsoft that would be an interesting thing to bring up to the people who create the compiler. om nom nom's answer below suggests they fixed a few things up that at least gave a better error message. So maybe they already did? I'm not sure where I would go hunting for that information though. Or which mailing lists to scroll through — EdgeCaseBerg, Jul 22 '15 at 19:17
Simply don't, and please get rid of that array. If you *really* have to have that image in your source *code* (notice something :P), you might want to try Base64-encoded strings. — Clashsoft, Jul 22 '15 at 19:20
thanks for the suggestion @Clashsoft hadn't thought to use Base64 for that. — EdgeCaseBerg, Jul 22 '15 at 19:31

score 7 · Accepted Answer · edited May 23 '17 at 12:34

7

Well I tried to reproduce your problem and newer scalac (2.11.7) has a better error message (hope it clears up the problem):

» scalac Arrays.scala
Arrays.scala:1: error: Could not write class testCase because it exceeds JVM code size limits. Method <init>'s code too large!
class testCase()  {
      ^
one error found

So it looks like, just like @Marius noted in comments, you're hitting "64K bytecode per method" limit imposed by JVM.

edited May 23 '17 at 12:34

Community

1
1

answered Jul 22 '15 at 19:15

om-nom-nom

62,329
13
183
228

So it's looking like an answer to how big anything can be depends on those JVM settings? Which look like some of the limits are reference in this question: http://stackoverflow.com/questions/17422480/maximum-size-of-a-method-in-java-7-and-8 – EdgeCaseBerg Jul 22 '15 at 19:16
Thanks @om-nom-nom this answers the question, but for clarity of an answer for anyone else who runs into this, could you edit your answer to explicitly state that it's a JVM code size limit as well as showing the error from the compiler? – EdgeCaseBerg Jul 22 '15 at 19:20

Why does a large array constructor call break the Scala compiler?

1 Answers1