Java heap memory issue and CPU goes high

Question

We are using Java as middleware to communicate with db - but nowadays we are getting CPU high around 400%. We have installed the JVM to monitor the issue - and found this error - could you please suggest us to improve this? I have seen these,

1. [Quick solution] Increase memory (my current server having 16GB ram, quad-core processor) -- any suggestion how much need to increase?

2. What are the objects in the application that occupy large portions of the heap? -- how to identify this as very basic knowledge on Java, is there any tool which would help me to identify this?

3. In which parts of the source code are these objects being allocated? -- how to identify this as very basic knowledge on Java, is there any tool which would help me to identify this?

   o

rg.glassfish.jersey.server.ContainerException: java.lang.OutOfMemoryError: GC overhead limit exceeded …httpserver.GrizzlyHttpContainer$ResponseWriter.rethrow(GrizzlyHttpContainer.java:318) …httpserver.GrizzlyHttpContainer$ResponseWriter.failure(GrizzlyHttpContainer.java:300) …lassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:486) org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:317) org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) org.glassfish.jersey.internal.Errors.process(Errors.java:315) org.glassfish.jersey.internal.Errors.process(Errors.java:297) org.glassfish.jersey.internal.Errors.process(Errors.java:267) …ssfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:292) org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1139) …ersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:375) org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) …sh.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) …sfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) java.lang.Thread.run(Thread.java:748) caused by java.lang.OutOfMemoryError: GC overhead limit exceeded

Answer 1

These kinds of mistakes are terrifying, because to find the real cause is complicated, and a intermediate java knowledge is required.

It never hurts a bit of documentation:

• https://www.baeldung.com/java-gc-overhead-limit-exceeded
• https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks002.html

Sudden change

If it is a sudden change, this may be the cause. If you have a intermediate control and monitor of your infrastructure, you should be able to detect when this problem started:

• after the release of a new version of java application
• after an upgrade of web server
• after an upgrade of the operative system
• after new version of database (e.g oracle 11 > 12)
• after low level changes in the database(ram, disk, etc)
• more users than expected. Maybe concurrent users

If some of the previous causes sounds familiar to you, just revert it and validate if is the cause of a resource (memory) exhaustion.

Legacy

In some cases, legacy servers or applications (installed several years ago) could be the cause of these kind of problems:

• temporary file are not deleted
• the disk reached its maximum
• deprecated java libraries

Try to upgrade the web server and deploy again your application. Also if your application is agnostic, try to deploy to another servers just to try.

A huge number of rows

Some times [GC overhead limit exceeded] is related to this error:

java.lang.OutOfMemoryError: Java heap space

Which is related with ResultSets that have a large number of rows or large values. As a result it can not allocate heap space in the JVM for the memory required.

If this is the case, try with:

stmt.setFetchSize

Sources:

• http://benjchristensen.com/2008/05/27/mysql-jdbc-memory-usage-on-large-resultset/
• https://stackoverflow.com/a/15559710/3957754
• https://docs.oracle.com/cd/A87860_01/doc/java.817/a83724/resltse5.htm
• https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#setFetchSize(int)

Source Code Manual Inspection

This snippet is able to replicate your error:

Map<Integer, String> dataMap = new HashMap<>();
Random r = new Random();
while (true) {
    dataMap.put(r.nextInt(), String.valueOf(r.nextInt()));
}

Try to review class by class in order to find blocks like the snippet.

Monitor Tools

There are several tools. Its usage is not so easy. I advice you choose one and test it with a simple application like:

• swing app
• spring boot minimal rest api
• web deployed in tomcat

If you are able to understand how it works, use it in your real application.

Visual VM

As you can see, this tool is able to scan all your classes and get relevant metadata like: live instances, memory, etc

Sources:

• https://visualvm.github.io/
• https://docs.oracle.com/javase/8/docs/technotes/guides/visualvm/snapshots.html

Java Mission Control

Java Flight Recorder and JDK Mission Control together create a complete tool chain to continuously collect low level and detailed runtime information enabling after-the-fact incident analysis

Sources:

• https://www.oracle.com/java/technologies/jdk-mission-control.html
• https://www.baeldung.com/java-flight-recorder-monitoring#3-visualize-data

Eclipse Profiler

In almost all java web servers, remote debug is a feature.

If you can configure it, you could try the eclipse profiles. You will see all the classes of your application:

Sources:

• https://www.yourkit.com/docs/java/help/eclipse.jsp
• Profiling a Java Application in Eclipse? (plug-in)
• https://marketplace.eclipse.org/taxonomy/term/13%2C31/favorites
• https://www.baeldung.com/java-profilers