14

I have a system in Scala, with a lot of simultaneous threads and system calls. This system has some problem, because memory usage is increasing over time.

The image bellow shows the memory usage for one day. When it gets to the limit, the process shuts down and I put a watch-dog to recover it again. inserir a descrição da imagem aqui

I periodically run the command

jcmd <pid> GC.run

And this makes the memory to increase slowly, but the leak still happens.

I analysed with jvisualvm, comparing to distinct moments in time, with 40 minutes delta. The image bellow shows the comparison between these two moments in time. Notice that there is an increase for instances of some classes like ConcurrentHashMap$HashEntry, SNode, WeakReference, char[] and String and many classes in the package scala.collection.concurrent.

memory leaked ojects

What can be causing the memory leak?

Edit 1: Investigating JVisualVM, I noticed object of CNode and INode classes that are in TriedMap, that is instanced inside sbt.TrapExit$App class. Here is the object hierarchy figure:

object hierarchy

Daniel Cukier
  • 11,502
  • 15
  • 68
  • 123
  • 1
    http://stackoverflow.com/questions/1218872/avoiding-scala-memory-leaks-scala-constructors?rq=1 http://stackoverflow.com/questions/7944148/weakreference-and-memory-leaks http://stackoverflow.com/questions/3871960/long-lived-java-weakreferences – Tony Sep 14 '14 at 18:36
  • I see list.toStream.map, but where do the scala.collection.concurrent come from? When you say "lots of threads" do you mean you "foo.par" a lot? I'm not asking as an expert in parallel collections. Are you using a TrieMap explicitly? – som-snytt Sep 14 '14 at 20:29
  • Lots of threads I mean there are hundreds of actors, each actor does system calls and create some futures to asynchronously execute tasks. I do not use TrieMap explicitly. – Daniel Cukier Sep 15 '14 at 16:55
  • Do you have updated answer? I also have this similar problem. Hope anybody can figure out root cause. Thanks in advance. – Haimei Jan 20 '16 at 21:13

2 Answers2

3

First capture a heap dump when your application crashes due to an out of memory issue. Add the following flags when starting the jvm

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/dump

Next you need to analyze the heap dump to figure out the source of the memory leak. I recommend using Eclipse MAT. The Leak Suspects report should give you a sense of what objects are actually causing the leak.

rrevo
  • 1,647
  • 1
  • 16
  • 22
  • With the heap diffs, I already have the idea of what is the source of the leaking. ConcurrentHashMap$HashEntry are increasing numbers. But I don't use these objects in my code, so it looks like something inside the scala or jvm – Daniel Cukier Sep 30 '14 at 15:04
  • You might not be directly instantiating the objects but it does happen. So you need to trace back as to what/when/where these objects are created. So who creates the CNode instance etc. Some object keeps a reference to the CNode preventing it (and the objects it references) from being garbage collected. Eclipse MAT has this view - http://help.eclipse.org/luna/index.jsp?topic=%2Forg.eclipse.mat.ui.help%2Ftasks%2Ffindingresponsibleobjects.html that should give you some information. – rrevo Oct 03 '14 at 21:59
0

Without seeing the implementation its hard to say. The title of your post suggests that there is a memory leak in Scala, but did you check your implementation against problems with releasing objects?

Did you check following:

  • Do you limit number of actors at all?
  • Do you set timeouts for the system calls?
  • Do you allow the actors to be removed from Heap when they performed therir tasks?
  • Did you count how many actors can fit into your memory or you are just creating "hundreds of actors" with hope that jvm will know "what to do"

What I'm trying to say is that maybe you run out of memory because you simply create to many objects which are not later released, because either they are still performing their tasks (no timeout) or you have created to many of them.

Maybe you need to scale your application to many jvms? How many jvms do you use?

walkeros
  • 4,736
  • 4
  • 35
  • 47