How to prevent Scala Futures from creating a memory leak

Question

I have a large collection of parallel processes that run. It seems that the mere creation of the parallel scala Futures creates a memory leak.

Example code below. Set the VM flag depending on your machine so it doesn't dump right at start in case you have many cores, "-Xmx100m -XX:+HeapDumpOnOutOfMemoryError".

import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Await
import scala.concurrent.duration._

object Bug extends App {
    println("Memory leak")
    while(true){
        Future {
            val data = new Array[Byte](1000000*1) // 1 MB
            println(".")
        }
    }
}

After a few minutes this code will slow significantly. A heap dump analysis reveals that the array scala.concurrent.forkjoin.ForkJoinTask[] seems to overflow.

The following link seems related but does not really offer any solution to the issue: https://issues.scala-lang.org/browse/SI-7336

java version "1.8.0_66" and Scala code runner version 2.11.7 — calloc_org, Jan 01 '16 at 11:52
At first I thought it may be the `println` blocking your Futures which would eventually cause a back-up and subsequent out-of-memory exception. But without the println I still see the same result you mentioned. I'm unconvinced this proves there is a leak though. **If** the JVM is faster at spawning Futures than at releasing/re-using memory there wouldn't be a leak but the same exception would result... — Ramón J Romero y Vigil, Jan 01 '16 at 12:07
@RamonJRomeroyVigil The spawning is an excellent point. I will run a longer test with a wait and report on the outcome. In my application I have some threads pushing data into a database, so the spawning issue may in fact be the root cause as they may be faster created than the DB can take the data up. Should be able to report in a few hours. — calloc_org, Jan 01 '16 at 12:25
@RamonJRomeroyVigil I think you identified the issue in this, it seems to be related to the creation. Running it with a small wait at the front of the future resolves the issue and the memory stays constant long term. Thanks!!! Related question: http://stackoverflow.com/questions/30178378/what-is-the-cause-of-this-strange-scala-memory-leak — calloc_org, Jan 01 '16 at 12:49

Maxim · Accepted Answer · 2016-01-01T13:20:35.630

What's happening here is that you're allocating memory much faster then GC`ing it.

In each execution of the while loop you're putting a task to the thread pool to allocate 1MB, you're allocating memory very fast.

After a while GC is what's slowing down.

Solution? Allocate memory slower.

BTW, it will behave in a similar way even if you don't declare the array. When you're running a Future you're submitting a task to the queue of the thread pool so the memory will increase very fast anyway.

To see it yourself you can run jstat (http://www.cubrid.org/blog/dev-platform/how-to-monitor-java-garbage-collection/)

jstat -gc $pid 1000

Here's a sample from my laptop, look how the GCT column (The total accumulated time for GC operations) is increasing

➜  ~  jstat -gc 24504 1000
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
35328.0 35840.0 22962.9  0.0   275968.0   0.0     395264.0   383080.0  11904.0 11387.3 1664.0 1545.4     30    1.333   3      2.125    3.458
35328.0 35840.0 22962.9  0.0   275968.0   0.0     395264.0   383080.0  11904.0 11387.3 1664.0 1545.4     30    1.333   3      2.125    3.458
35328.0 35840.0 22962.9  0.0   275968.0   0.0     395264.0   383080.0  11904.0 11387.3 1664.0 1545.4     30    1.333   3      2.125    3.458
36352.0 36864.0 8148.0  0.0   275456.0   0.0     562688.0   533891.1  11904.0 11407.6 1664.0 1548.2     36    1.772   4      4.751    6.524
36352.0 36864.0 8148.0  0.0   275456.0   0.0     562688.0   533891.1  11904.0 11407.6 1664.0 1548.2     36    1.772   4      4.751    6.524
36352.0 36864.0  0.0    0.0   275456.0 143402.5  699392.0   486444.1  11904.0 11407.6 1664.0 1548.2     37    1.772   4      6.867    8.640
32768.0 33792.0 22498.3 11986.9 278528.0 278251.8  699392.0   632212.1  11904.0 11437.5 1664.0 1549.8     47    2.548   4      6.867    9.415
29696.0 31232.0 20498.9  0.0   284160.0   0.0     699392.0   678668.1  11904.0 11448.8 1664.0 1551.3     50    2.768   5      6.867    9.635
29696.0 31232.0 20498.9  0.0   284160.0   0.0     699392.0   678668.1  11904.0 11448.8 1664.0 1551.3     50    2.768   5      6.867    9.635
29696.0 31232.0 20498.9  0.0   284160.0   0.0     699392.0   678668.1  11904.0 11448.8 1664.0 1551.3     50    2.768   5      6.867    9.635
29696.0 31232.0  0.0    0.0   284160.0 283932.1  699392.0   691190.3  11904.0 11448.8 1664.0 1551.3     50    2.768   6     10.452   13.220
29696.0 31232.0  0.0    0.0   284160.0 283932.1  699392.0   691190.3  11904.0 11448.8 1664.0 1551.3     50    2.768   6     10.452   13.220
29696.0 31232.0  0.0    0.0   284160.0 283932.1  699392.0   691190.3  11904.0 11448.8 1664.0 1551.3     50    2.768   6     10.452   13.220
29696.0 31232.0  0.0    0.0   284160.0 283932.1  699392.0   691190.3  11904.0 11448.8 1664.0 1551.3     50    2.768   6     10.452   13.220
29696.0 31232.0  0.0    0.0   284160.0 283932.1  699392.0   691190.3  11904.0 11448.8 1664.0 1551.3     50    2.768   6     10.452   13.220
29696.0 31232.0  0.0    0.0   284160.0 283918.2  699392.0   698585.9  11904.0 11451.8 1664.0 1551.3     50    2.768   7     15.069   17.837
29696.0 31232.0  0.0    0.0   284160.0 283918.2  699392.0   698585.9  11904.0 11451.8 1664.0 1551.3     50    2.768   7     15.069   17.837
29696.0 31232.0  0.0    0.0   284160.0 283918.2  699392.0   698585.9  11904.0 11451.8 1664.0 1551.3     50    2.768   7     15.069   17.837
29696.0 31232.0  0.0    0.0   284160.0 283918.2  699392.0   698585.9  11904.0 11451.8 1664.0 1551.3     50    2.768   7     15.069   17.837
29696.0 31232.0  0.0    0.0   284160.0 16821.1   699392.0   699140.7  11904.0 11452.4 1664.0 1551.3     50    2.768   7     19.562   22.330
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699140.7  11904.0 11452.4 1664.0 1551.3     50    2.768   8     19.562   22.330
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699140.7  11904.0 11452.4 1664.0 1551.3     50    2.768   8     19.562   22.330
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699140.7  11904.0 11452.4 1664.0 1551.3     50    2.768   8     19.562   22.330
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699140.7  11904.0 11452.4 1664.0 1551.3     50    2.768   8     19.562   22.330
29696.0 31232.0  0.0    0.0   284160.0 284029.3  699392.0   699116.2  11904.0 11453.7 1664.0 1551.3     50    2.768   9     24.112   26.880
29696.0 31232.0  0.0    0.0   284160.0 284029.3  699392.0   699116.2  11904.0 11453.7 1664.0 1551.3     50    2.768   9     24.112   26.880
29696.0 31232.0  0.0    0.0   284160.0 284029.3  699392.0   699116.2  11904.0 11453.7 1664.0 1551.3     50    2.768   9     24.112   26.880
29696.0 31232.0  0.0    0.0   284160.0 284029.3  699392.0   699116.2  11904.0 11453.7 1664.0 1551.3     50    2.768   9     24.112   26.880
29696.0 31232.0  0.0    0.0   284160.0 284029.3  699392.0   699116.2  11904.0 11453.7 1664.0 1551.3     50    2.768   9     24.112   26.880
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699092.9  11904.0 11455.9 1664.0 1551.3     50    2.768  10     28.728   31.496
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699092.9  11904.0 11455.9 1664.0 1551.3     50    2.768  10     28.728   31.496
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699092.9  11904.0 11455.9 1664.0 1551.3     50    2.768  10     28.728   31.496
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699092.9  11904.0 11455.9 1664.0 1551.3     50    2.768  10     28.728   31.496
29696.0 31232.0  0.0    0.0   284160.0 176242.5  699392.0   699074.7  11904.0 11455.9 1664.0 1551.3     50    2.768  11     33.550   36.318
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699074.7  11904.0 11455.9 1664.0 1551.3     50    2.768  11     33.550   36.318
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699074.7  11904.0 11455.9 1664.0 1551.3     50    2.768  11     33.550   36.318
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699074.7  11904.0 11455.9 1664.0 1551.3     50    2.768  11     33.550   36.318
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699074.7  11904.0 11455.9 1664.0 1551.3     50    2.768  11     33.550   36.318
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699074.7  11904.0 11455.9 1664.0 1551.3     50    2.768  11     33.550   36.318
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699313.9  11904.0 11458.4 1664.0 1551.3     50    2.768  12     38.797   41.565
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699313.9  11904.0 11458.4 1664.0 1551.3     50    2.768  12     38.797   41.565
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699313.9  11904.0 11458.4 1664.0 1551.3     50    2.768  12     38.797   41.565
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699313.9  11904.0 11458.4 1664.0 1551.3     50    2.768  12     38.797   41.565
29696.0 31232.0  0.0    0.0   284160.0 284160.0  699392.0   699313.9  11904.0 11458.4 1664.0 1551.3     50    2.768  12     38.797   41.565

Verified, thanks. Credit also to @RamonJRomeroyVigil – calloc_org Jan 01 '16 at 13:19 — calloc_org, Jan 01 '16 at 13:19

How to prevent Scala Futures from creating a memory leak

1 Answers1