Scala hit GC overhead when running large external process

Question

I have an external program which generate some data I need. Usually, I redirect its output to a file, then read it from my Scala application, e.g.

app.exe > output.data

Now, I want to integrate the process, so I did

val stream = "app.exe" lineStream
stream foreach { line => doWork(_) }

Unfortunately, I got GC overhead exception after a while. This app.exe may generate very large output files, e.g. over 100MB. So I think during the streaming, Scala has been creating/destroying the line string instance thousands of times, and cause the overhead.

I know I can tune the JVM variables to increase the GC overhead throttling. But I am looking for a way that it doesn't need to create a lot of small line instances.

I think you meant [GC overhead](http://stackoverflow.com/questions/1393486/error-java-lang-outofmemoryerror-gc-overhead-limit-exceeded). — Frédéric Dumont, Mar 10 '15 at 00:44

score 3 · Accepted Answer · answered Mar 10 '15 at 01:02

The problem is probably due to memoization, which is a side effect of foreach-ing over a stream this way. Effectively, you are rooting the whole file in memory.

See lots and lots of info on how to avoid this here: http://blog.dmitryleskov.com/programming/scala/stream-hygiene-i-avoiding-memory-leaks/

Specifically, you are violating rule #1. Try defining your stream as a def, not a val.

Scala hit GC overhead when running large external process

1 Answers1