2

We have server with 35gb memory and Intel® Xeon(R) E5-1620 0 @ 3.60GHz × 8 CPU. I am running a multithreaded program designed with akka actors and written in scala. In the program, there are 4 actors with tasks:

1) Lazy reading from file with Scala's BufferedSource and iterator,

2) Tokenizing sentences,

3) Calculating single and bigram words frequency for a given window size, and putting them into a map (one map for single words [String, Int], one for tuple words[WordTuple,Int),

4) Merging returned hasmaps into one hashmap and when all lines read from file and write them into a file.

My custom jvm settings is as follows:

-Xms34g

-Xmx34g

-XX:ReservedCodeCacheSize=240m

-XX:+UseParallelGC

-XX:ParallelGCThreads=4

-XX:NewSize=12g

-XX:SoftRefLRUPolicyMSPerMB=50

-ea

-Dsun.io.useCanonCaches=false

-Djava.net.preferIPv4Stack=true

-XX:+HeapDumpOnOutOfMemoryError

-XX:-OmitStackTraceInFastThrow

-Dawt.useSystemAAFontSettings=lcd

-Dsun.java2d.renderer=sun.java2d.marlin.MarlinRenderingEngine

-verbose:gc

-XX:+PrintGCDetails

-Xloggc:gc.log

My application.conf is as follows:

systemParameters {
  linesPerActor = 5
  windowSize = 6
  threadPoolSize = 5
}


akka.actor.deployment {

  /wordTokenizerRouter {
    router = round-robin-pool
    nr-of-instances = 5
  }

  /frequencyCalculatorRouter {
    router = round-robin-pool
    nr-of-instances = 5
  }
}

The problem:

I am processing a text file with size 15gb. Program starts working, after a while, say 2 hours, those tokenizing, calculating operations is almost not working, no operations can perform. The operation that takes 300 milliseconds starts taking 100000 seconds. But the cpu usage is %100 for all processors. I've tried using jvisualvm to motinor it but sampler is not working with this high cpu usage, so I could not identify which process is making cpu %100. I check gc activity from jvisualvm and most of the time it is using about %10 cpu. So, what could be the problem with my program, which process is possibly using all the cpu?

Here some screenshots from jvisualvm when operations in the program is stop but cpu usage is %100:

Garbage collector status screenshot

Overall status screenshot

Hope I explained it clear. Thanks in advance for your answers.

katilsperm
  • 61
  • 1
  • 5

1 Answers1

0

I would look into a few areas.

  1. Your heap looks full, including the old generation. Another hint: Of the 8 hours 20 minutes runtime, your application spent 5 hours and 45 minutes in olg generation GC. As you're heap is getting full, it triggers full GCs, one after another. With the Parallel GC, this will use all cores during the full GC. Look at you're gc.log, and see how many times the full GC gets triggered.
  2. During the CPU loads, create a few thread dumps. You can use VisualVM' or the `jstack' command. In Visual VM its on the 'Threads' tab, 'Thread Dump'. Look at the stack dumps and look for the 'Runnable' threads, which are not at some blocking / IO API. See what they are doing.

If it's spending time in garbage collection. I would take a heap dump and analyze what memory is held. You can take a heap dump also in VisualVm in the Monitor tab and do a rough analysis there.

Gamlor
  • 12,978
  • 7
  • 43
  • 70
  • As I observed now, as time passes, the frequency of garbage collections are increasing such that 5 seconds running application is followed by 10 seconds garbage collection. And this time period is getting narrow as time passes. Now, I observed more specific problem, garbage collector runs are cannot cleaning the whole space, it is just cleaning a little bit of unused references and application heap is getting full again. Here are the proof of it with a screenshot: http://imgur.com/0NbZelt and another for gc using high cpu: http://imgur.com/a/YSrsw so, application processes is working less. – katilsperm Apr 26 '17 at 14:39
  • What could be the reason of garbage collection cannot clean all unused references when it is running? – katilsperm Apr 26 '17 at 14:44
  • Your application holds references to objects, so GC cannot collect is. Find out out what holds those reference if you really need it. Like, take a heap dump. So what instances are in the heap. Use 'find roots' to find out how the reference is held alive. – Gamlor Apr 26 '17 at 23:42