Garbage collection with scala future

Question

I have a bunch of xml files that I'm trying to process in parallel. My scala code (2.9.2) using future starts out fine but ends up eating up nearly 100% of the 32G I have on my machine. This doesn't happen when I do this sequentially so my guess is there's something wrong with garbage collection in using scala futures.

Here's a stripped down version of my code. Can anyone tell me what's wrong?

val filenameGroups = someStringListOfFilepaths.grouped(1000).toStream
val tasks = filenameGroups.map {
  fg =>
    scala.actors.Futures.future {
      val parser = new nu.xom.Builder() // I'm using nu.xom. Not sure it matters.
      fg.map {
        path => {
          val doc = parser.build(new java.io.File(path))
          val result = doc.query(some xpath query)
          result
        }
      }.toList
    }
}

val pairs = tasks.par.flatMap(_.apply)

ETA: Ok, I solved this but I still have no idea why this makes a difference.

I abstracted most of the code in the inner loops then reran it. And pulled out the parser instantiation from future. Memory usage now stays flat at a decent 17%. Does anybody have any idea why this would make a difference?

Here's a simplified version of what I did:

def process(arglist...) = yada

val tasks = filenameGroups.map {
  fg =>
    val parser = new nu.xom.Builder()
    scala.actors.Futures.future {
      process(fg, parser)
    }
}

val pairs = tasks.par.flatMap(_.apply)

Answer 1

Futures can't really predict how many threads you want or how much memory your computations will take, so it's generally your responsibility to put appropriately serialized computations inside a modest number of futures. In particular, if you are on an 8 core machine, you probably don't want to group much smaller than someStringListOfFilepaths.length/8 (less if your files are so large that you can't have 8 in memory at once). You can use the standard Java trick of inspecting the number of cores, covered on SO and many other places, if you want to scale it per machine without having to think about it. (Might want to inspect Runtime.getRuntime.maxMemory also in that case, just in case you're on a machine with lots of cores and not much RAM (or not much allocated for the VM).)

(Incidentally, in your minimal example there's both laziness and futures, but the laziness doesn't do anything for you. Futures are already not running when created, so delaying the instantiation of the futures probably doesn't help you any.)

Also, note that if you have 200k files, you will end up with 200k results, and depending on how large a result is, that could eat up a lot of memory. Probably not 32G, but who knows what's in the files?