Java 8 with Jetty on linux memory issue

Question

Can you please help me resolving the following issue:

Context:

We are trying to migrate our existing application which is currently running on Java6( on Glassfish) in production to Java8 (on Jetty9) setup. Earlier, We were able to successfully migrate the same setup on Java7(on jetty9). But the customer has decided to go with Java 8 now. In this process we have encountered some memory issues and below are the details:

Problem Description:

After starting the Jetty server, initial (RES)memory usage of java process is around 5.5g. After running the application for sometime the memory usage slowly shoots-up and consumes maximum available physical memory(8g) on the machine, eventually crashing the server/system.

This issue is encountered only on the linux environment. No such issues are found on Windows environment.

Profiler findings:

Monitored the server with VisualVM and jconsole. In both the profilers JVM's memory(heap & non-heap) usage is under the allocated limits.

Environment details:

Java Version :    8
Server       :     Jetty 9.2.10
OS           :    linux on a virtual machine(Linux version 2.6.32-279.14.1.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Tue Nov 6 23:43:09 UTC 2012
Java Options :        
        -Xms3072M

        -server

        -XX:+UnlockDiagnosticVMOptions

        -XX:+LogVMOutput

        -XX:+UseG1GC

        -XX:MaxGCPauseMillis=75

        -Xmx3072M

        -Xss1024K

        -XX:InitialCodeCacheSize=192M

        -XX:CodeCacheExpansionSize=3M

        -XX:CodeCacheMinimumFreeSpace=3M

        -XX:ReservedCodeCacheSize=600M

        -XX:MinMetaspaceExpansion=3M

        -XX:MaxMetaspaceExpansion=18M

        -XX:MaxMetaspaceSize=500M

        -XX:MaxDirectMemorySize=288M

        -XX:CompressedClassSpaceSize=512M

        -XX:ParallelGCThreads=12

        -XX:ConcGCThreads=4

        -Dsun.rmi.dgc.server.gcInterval=86400000

        -Dsun.rmi.dgc.client.gcInterval=86400000

PS: Please don't mark this as duplicate. I have read many answers on stackoverflow, but nothing addressed or solved my issue.

Update

I started jetty with the following Java Options, and since then the memory usage is between 4.5g-4.8g (approximately 142 hours). It looks stable to me. I reserved around 2g of memory through java option flags(Xmx & MetaspaceSize), but additional 2.5g is used always. Is this normal behavior for Java 8 on a linux machine?

Java Options used:

-server
-XX:+UnlockDiagnosticVMOptions
-XX:+LogVMOutput
-XX:LogFile=../logs/jvm.log

-XX:+UseG1GC
-XX:MaxGCPauseMillis=75
-XX:ParallelGCThreads=12
-XX:ConcGCThreads=12
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-Xloggc:../logs/gc.log
-XX:NativeMemoryTracking=summary

-Xmx1500m
-Xss256k

-XX:MaxMetaspaceSize=512m

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/opt/logs/jetty.hprof

-Dsun.rmi.dgc.server.gcInterval=86400000
-Dsun.rmi.dgc.client.gcInterval=86400000

Thanks for your time!

Are you using native code somewhere (explicitly, almost every java program uses lots of native code under the hood)? Like native bindings to openssl, or a http connector like http://netty.io/wiki/native-transports.html ? Memory leaks in there don't respect the JVM limits since the memory isn't managed by the JVM. http://www.evanjones.ca/java-native-leak-bug.html has some hints how to look for those problems — zapl, Oct 13 '15 at 14:17
@zapl thanks for your inputs! I will check for native bindings in my project. — Diablo, Oct 13 '15 at 14:27
I think `MaxDirectMemorySize` does not apply to mmaped files, only to `ByteBuffer.allocateDirect()`, so inspect your heap dump for instances of `DirectByteBuffer` and tally up their capacity. — the8472, Oct 13 '15 at 18:43

Bruno Ranschaert · Answer 1 · 2016-12-01T16:33:17.573

It seems to be an issue in Java8/9 that manifests itself in Jetty due to the annotations module that scans the jars and has a memory leak. Ref. https://github.com/eclipse/jetty.project/issues/575. A solution for me (because I do not use Jetty annotations) is to disable the annotations module by commenting out the lines in jetty/modules/annotations.mod. So tie file looks like this:

#
# Jetty Annotation Scanning Module
#

[depend]
# Annotations needs plus, and jndi features
plus

[lib]
# Annotations needs jetty annotation jars
lib/jetty-annotations-${jetty.version}.jar
# Need annotation processing jars too
#lib/annotations/*.jar

[xml]
# Enable annotation scanning webapp configurations
#etc/jetty-annotations.xml

Edit 1 - Alternative solution

Switching off all annotation scanning can be too drastic, it turns of jsp as well because it is dependent. The alternative is to provide a web application context which restricts the scanning using a pattern. Save this in an xml and deploy it in the webapps together with the war or include it in the war.

<Configure class="org.eclipse.jetty.webapp.WebAppContext">
   <Set name="contextPath">/[myApp]</Set>
   <Set name="war">/[DIRECTORY]/[myApp[.war</Set>
   <Call name="setAttribute">
      <Arg>org.eclipse.jetty.server.webapp.WebInfIncludeJarPattern</Arg>
      <Arg>SCAN-NO-JARS</Arg>
    </Call>
</Configure>

I was struggling with this myself, see related issue https://stackoverflow.com/questions/40807646/jetty-9-java-8u74-always-allocates-maximum-xmx-memory-on-linux. I initially thought it was a pure Java issue. — Bruno Ranschaert, Nov 29 '16 at 16:43

Java 8 with Jetty on linux memory issue

1 Answers1

Linked