19

My project has started using java 8 from java 7.

After switching to java 8, we are seeing issues like the memory consumed is getting higher with time.

Here are the investigations that we have done :

  • Issues comes only after migrating from java7 and from java8
  • As metaspace is the only thing related to memory which is changes from hava 7 to java 8. We monitored metaspace and this does not grow more then 20 MB.
  • Heap also remains consistent.

Now the only path left is to analyze how the memory gets distributes to process in java 7 and java 8, specifically private byte memory. Any thoughts or links here would be appreciated.

NOTE: this javaw application is a swing based application.

UPDATE 1 : After analyzing the native memory with NMT tool and generated a diff of memory occupied as compare to baseline. We found that the heap remained same but threads are leaking all this memory. So as no change in Heap, I am assuming that this leak is because of native code.

So challenge remains still open. Any thoughts on how to analyze the memory occupied by all the threads will be helpful here. Below are the snapshots taken from native memory tracking.

In this pic, you can see that 88 MB got increased in threads. Where arena and resource handle count had increased a lot.

enter image description here

in this picture you can see that 73 MB had increased in this Malloc. But no method name is shown here. enter image description here

So please throw some info in understanding these 2 screenshot.

Onki
  • 1,879
  • 6
  • 38
  • 58
  • 10
    What kind of memory consumption? Is the heap growing? Is more native memory being taken? (That's where some of the PermGen was moved). The question is too general. You need to show some memory analyses. – RealSkeptic Sep 19 '15 at 09:32
  • @RealSkeptic The private bytes consumed are increasing with java 8. As of now we are monitoring private Bytes only. If there is any specific data that I need to collect please let me know, I ll look into that – Onki Sep 19 '15 at 09:36
  • 4
    Dump the heap and analyze it (e.g. with Eclipse Memory Analyzer) both for JDK7 and JDK8. Look for significant differences. Submit your findings. – Tagir Valeev Sep 19 '15 at 09:50
  • @TagirValeev I can tell you about private byte consumption as of now. With java 7 it is close to 100 MB and with java 8 it gets close to 300 MB. – Onki Sep 19 '15 at 10:01
  • 2
    If you *think* there's a problem you should actually take measurements to verify that there actually *is* a problem and to quantify it if there is. Not just the stats shown in the task manager but also those printed by various java diagnostic tools. – the8472 Sep 19 '15 at 14:01
  • 1
    [Probably connected](https://bugs.openjdk.java.net/browse/JDK-8046339), but more information like memory dump analysis is still necessary. – Tagir Valeev Sep 21 '15 at 08:30
  • Hi @Haj did you find a solution to this issue? I am facing a similar scenario where the application memory and native memory size is normal but the actual RAM usage is growing up with time. I am using java 8 with G1 garbage collector. – Anish Gupta Jul 09 '17 at 18:12
  • @AnishGupta I saw your msg, we have not yet found a solution, and still searching for it. Did you find any solution – Onki Sep 08 '17 at 08:49
  • @RealSkeptic I have updated the ans, Please see if you have any further inputs – Onki Sep 12 '17 at 09:26
  • @Tagir Valeev I have updated the ans, Please see if you have any further inputs – Onki Sep 12 '17 at 09:26
  • @the8472 I have updated the ans, Please see if you have any further inputs – Onki Sep 12 '17 at 09:26
  • Is it possible that memory consumption by direct byte-buffers grew after upgrading to Java8? (`MBeans/java.nio/BufferPool/direct/MemoryUsed` in jvisualvm) – yegodm Sep 12 '17 at 14:27
  • 1
    It might worth to check native memory usage with this tool https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html. – yegodm Sep 12 '17 at 15:12
  • @yegodm this link looks helpful. I will start investigating further and will udpate you – Onki Sep 14 '17 at 05:09
  • @yegodm I analyzed and updated the qsn with the results of NMT tool,, please check and share if you have any further updates – Onki Oct 24 '17 at 05:21
  • Now it becomes tricky. Further steps depend on the platform you are running your app on. First of all, you could probably check the bug database for similar problems. For example, these two - http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8180048 and http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8178304. – yegodm Oct 24 '17 at 10:43

5 Answers5

5

You may try another GC implementation like G1 introduced in Java 7 and probably the default GC in Java 9. To do so just launch your Java apps with:

-XX:+UseG1GC

There's also an interesting functionality with G1 GC in Java 8u20 that can look for duplicated Strings in the heap and "deduplicate" them (this only works if you activate G1, not with the default Java 8's GC).

-XX:+UseStringDeduplication

Be aware to test thoroughly your system before going to production with such a change!!!

Here you can find a nice description of the diferent GCs you can use

Sergio
  • 614
  • 5
  • 13
  • changing GC is not an option as this issue does not seem to be related to heap or GC. – Onki Sep 12 '17 at 10:23
2

consider optimising the JVM options

  1. Parallel Collector(throughput collector)

    -XX:+UseParallelGC

  2. concurrent collectors (low-latency collectors)

    -XX:+UseConcMarkSweepGC

  3. use String Duplicates remover

    -XX:+UseStringDeduplication

  4. optimise compact ratio

    -XXcompactRatio: and refer link1 link2

RamiReddy P
  • 1,628
  • 1
  • 18
  • 29
  • but why should we look into GC when heap consistently remains same. There seems to be no issue with GC or heap. – Onki Sep 14 '17 at 17:15
  • with Java 8 update 152 this issue does not seem to come.. and thank you very much for sharing inputs for narrowing down the investigation – Onki Feb 02 '18 at 11:11
2

In this my answer you can see information and references how to profile native memory of JVM to find memory leaks. Shortly, see this.

UPDATE

Did you use -XX:NativeMemoryTracking=detail option? The results are straightforward, they show that the most memory allocated by malloc. :) It's a little bit obviously. Your next step is to profile your application. To analyze native methods and Java I use (and we use on production) flame graphs with perf_events. Look at this blog post for a good start.

Note, that your memory increased for threads, likely your threads grow in application. Before perf I recommend analyze thread dumps before/after to check does Java threads number grow and why. Thread dumps you can get with jstack/jvisualvm/jmc, etc.

egorlitvinenko
  • 2,736
  • 2
  • 16
  • 36
  • there is good information given in your answer for which you have given the link. I would have to test with the info you provided and will update you again. – Onki Sep 15 '17 at 10:08
  • giving bounty to this answer as it contains most useful information. But still not accepting this as answer as I still have to use this information and test in my project. – Onki Sep 15 '17 at 10:12
  • If we will see results it would more likely that give concrete answer. Thank you, keeps up updated. – egorlitvinenko Sep 15 '17 at 10:41
  • @Onki updated. Sorry, for late response. Please, don't hesitate write me in message to take attention. – egorlitvinenko Oct 31 '17 at 09:14
  • few other things which is against analyzing thread dump is that thread count remains same n its just the memory occupies. Another is with respect to profiling, why we are not thinking about this because this issue is only observed with Java later then 7u45.. Just see if other thoughts strikes your mind – Onki Nov 06 '17 at 07:48
  • @Onki if you sure that the difference appears only starting from 7u45 then just check change log and try to find what could be related. – egorlitvinenko Nov 15 '17 at 17:34
  • yes I am sure that error comes up after 7u45. can you give more info about change log – Onki Nov 28 '17 at 11:42
  • Just google it - http://www.oracle.com/technetwork/java/javase/7u45-relnotes-2016950.html – egorlitvinenko Nov 28 '17 at 14:43
  • with Java 8 update 152 this issue does not seem to come.. and thank you very much for sharing inputs for narrowing down the investigation – Onki Feb 02 '18 at 11:11
2

I encountered the exact same issue.

Heap usage constant, only metaspace increase, NMT diffs showed a slow but steady leak in the memory used by threads specifically in the arena allocation. I had tried to fix it by setting the MALLOC_ARENAS_MAX=1 env var but that was not fruitful. Profiling native memory allocation with jemalloc/jeprof showed no leakage that could be attributed to client code, pointing instead to a JDK issue as the only smoking gun there was the memory leak due to malloc calls which, in theory, should be from JVM code.

Like you, I found that upgrading the JDK fixed the problem. The reason I am posting an answer here is because I know the reason it fixes the issue - it's a JDK bug that was fixed in JDK8 u152: https://bugs.openjdk.java.net/browse/JDK-8164293

The bug report mentions Class/malloc increase, not Thread/arena, but a bit further down one of the comments clarifies that the bug reproduction clearly shows increase in Thread/arena.

ishaaq
  • 6,329
  • 3
  • 16
  • 28
1

This issue does not come with Java 8 update 152. The exact root cause of why it was coming with earlier versions is still not clearly identified.

Onki
  • 1,879
  • 6
  • 38
  • 58