0

We have two EC2 instances running a web app in an embedded Jetty (version 9.3.8.v20160314). The web app receives requests from mobile app and after some processing, forwards them to a third party server. Peak load is around 10 requests per second. We use Hibernate- 4.2.4.Final, Java 1.8.0_25, hibernate-c3p0 -4.2.1.Final, JAX-RS resteasy-jaxrs-3.0.7.Final with MariaDB as DB.

The issue which is recurring every 7 to 10 days in both the EC2 instances (approximately at the same time) is OOME: Compressed class space. The CPU utilization goes above 70% and both the instances become unresponsive. From application the remote hosts could not be connected, but ping works fine. Restart of the applications resolves the issue.

Server Log: com.sun.xml.ws.server.sei.EndpointMethodHandler invoke SEVERE: Compressed class space java.lang.OutOfMemoryError: Compressed class space

Application Log: com.amazonaws.http.AmazonHttpClient - Unable to execute HTTP request: Broken pipe java.net.SocketException: Broken pipe

com.amazonaws.http.AmazonHttpClient - Unable to execute HTTP request: Remote host closed connection during handshake

The issue did not occur while using Java 1.7 but only when moved to Java 1.8. JVM 1.8 runs with Xmx4000m. Thread dump shows 41% threads waiting on monitor in c3pO thread pool as below: Thread dump trace

Heap dump analysis using Eclipse MAT gets stuck while computing dominator tree, but memory leaks estimate using HPJmeter gives below suspects: HPJmeter Memory leak suspects

Also, Heap size: 243,354,984 Classes: 1,800,395 Objects: 2,592,397

Questions:

1) Why OOME: Compressed class space exception occurring and how to identify leak suspect?

2) Are sockets not garbage collected or they are insufficient for new connections? Same for heap size, should heap size be increased?

3) Is there any connection leak in EntityManager utility using c3pO pool?

4) Do c3pO properties need to be modified to reclaim free sockets? Ideally, c3pO should withstand network outages.

5) Why the issue did not occur in Java 1.7 and occurs in Java 1.8?

Thanks in advance.

espkay
  • 11
  • 5
  • Are you saying you have 1.8 million classes, you have only 4 GB of memory. Note: compressed class pointers are limited in terms of how much space they can use. – Peter Lawrey Oct 07 '16 at 12:20
  • Possible duplicate http://stackoverflow.com/questions/30511439/java-lang-outofmemoryerror-compressed-class-space – Peter Lawrey Oct 07 '16 at 12:20
  • sounds like something is leaking classes or classloaders – the8472 Oct 07 '16 at 14:35

2 Answers2

3

Each object has a pointer (_klass) to VM metadata class. This pointer is also compressed on 64-bit JVM. In JDK 8, permanent generation was removed and replaced by metaspace memory. This metaspace is dynamically growing memory and can reach native memory limit if the VM flags are not configured to control this metaspace memory. Prior Java 8, permanent memory is used and its size is fixed with VM flags. It can’t change its size at run time. When compared with heap memory space, this permanent memory space (non-heap) is very small and will usually be within 4GB limit to store class metadata and other metadata. So the virtual memory address range can be 0 to 4 GB if the zero-based virtual memory is assigned by operating system to this permanent memory. So compressing the memory addresses of klass objects using only 32-bits in this permanent memory location is not a problem.

But in Java 8, permanent memory is replaced with metaspace in order to support some performance improvements. This memory location can grow till it reaches the native memory limit if -XX: MaxMetaspaceSize VM is not set to control this growth. If it can grow till the native memory limit, then the klass objects might also be stored at any higher memory boundaries. This will be a problem to compress that memory location using 32-bit offset. So in order handle this problem and allocate a separate memory space for klass instances alone, Java introduced a new memory space called “compressed class space” in Java 8 version. The inclusion of compressed class space memory in Java 8 introduced “Java.lang.OutOfMemoryError: Compressed class space”. This OutOfMemoryError is triggered when the compressed class space memory reaches its maximum default capacity of 1 GB and there are no more room to store additional klass instances.

In order to know more details about this OutOfMemoryError and about klass filed, you can read these books.

  1. Java Performance Optimization: Compressed OOPS
  2. Java Performance Optimization: How to avoid the 10 OutOfMemoryErrors
Nirmal
  • 89
  • 6
1

Why OOME: Compressed class space exception occurring and how to identify leak suspect?

I means you have too many classes loaded and you are using compressed class pointers which means that if you have 1.8 million classes you are going to run out of space.

Are sockets not garbage collected or they are insufficient for new connections? Same for heap size, should heap size be increased?

Sockets are garbage collected. You should really close them to avoid running out of resources but that not the problem you are having.

Why the issue did not occur in Java 1.7 and occurs in Java 1.8?

Java 1.7 has perm gen which works differently and I suspect it didn't have compressed class pointers.

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130