16

We are running on java-8-oracle.

We moved to java8 six month ago.

In the past few days we have been getting an OOME from time to time and we haven't been able to identify or reproduce the problem.

When we execute a call to the server (tomcat) we get this error on the stacktrace:

java.lang.OutOfMemoryError: Compressed class space

Restarting the server solves the problem. The same call to other server works, and so does another call of another type to the same server.

When looking on the gc.log we see:

2015-05-27T16:05:42.991+0000: 98774.440: [Full GC (Last ditch collection) 98774.440: [CMS: 575745K->575330K(3495936K), 0.8687777 secs] 575745K->575330K(4107008K), [Metaspace: 97940K->97940K(1396736K)], 0.8696093 secs] [Times: user=0.95 sys=0.00, real=0.88 secs]
2015-05-27T16:05:55.486+0000: 98786.935: [Full GC (Metadata GC Threshold) 98786.935: [CMS: 573414K->578735K(3495936K), 0.9372859 secs] 925046K->578735K(4107008K), [Metaspace: 99428K->99428K(1396736K)], 0.9386626 secs] [Times: user=1.01 sys=0.00, real=0.94 secs]

jstat -gc returns:

 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
87296.0 87296.0  0.0   3151.4 523776.0 148284.4 3495936.0   574868.5  1395640.0 98066.3 1048576.0 11339.1  12165  636.851  223   116.957  

753.808

I don't see any memory problems either in the jstat log or in the gc log.

Trying to run jmap -clstats hangs:

Attaching to process ID 5110, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.25-b02
finding class loader instances ..
assylias
  • 321,522
  • 82
  • 660
  • 783
user1236097
  • 193
  • 1
  • 2
  • 8
  • With which Xms and Xmx switches do you launch the JVM ? I recommend you use visualvm or a similar tool to better see and learn how your JVM sizes are set up. Or use Eclipse Memory Analyzer. For the time being you can try to increase the compressed classes space with -XX: CompressedClassSpaceSize. To better analyze the problem you should set the JVM to heap dump on OOME. – Marged May 28 '15 at 16:06
  • After what time period do you see this exception? Did you try increasing the CompressedClassSpace? e.g: -XX: CompressedClassSpaceSize=1g ? If you see the problem again but after a longer time period you seem to have a memory leak of some sort. – David George May 28 '15 at 16:07
  • @DavidG - first time encounter 2 days ago in one of the server (we didnt deploy new version). Restart server and than saw it again after 12 hours in only one of the servers. Stress load dosnt help to reproduce. The compress size is stay almost the same and it not closer to 1G which is the default. – user1236097 May 28 '15 at 18:34
  • 1
    This might be a JVM bug. There's an ongoing [hotspot-gc-dev discussion](http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-May/013420.html) of a problem with the similar symptoms. Try [tuning](http://stackoverflow.com/a/25252244/3448419) the Metaspace grow-shrink policy. – apangin May 29 '15 at 19:04

3 Answers3

15

We faced a similar issue. Unfortunately heapdumps won't help you since the classes are not in the heap but in native memory. Enable these in your JVM settings to troubleshoot the classes loaded:

-XX:+PrintGCDetails -XX:+TraceClassUnloading -XX:+TraceClassLoading

In our case the issue was JAXBContext.newInstance not being singleton.

Good luck, Albert

Hearen
  • 7,420
  • 4
  • 53
  • 63
Albert
  • 159
  • 1
  • 2
7

With compressed oops and compressed class pointers the available space for classes is constrained due to the necessary pointer mangling. 1GB in your case.

That's a lot of classes, so this might be an indicate that something in your application is creating a lot of classes and never releasing them. Application reload maybe?

If you are certain that your application just needs that much memory for classes you can try bumping the limit via -XX:CompressedClassSpaceSize=... or disabling compressed class pointers via -XX:-UseCompressedClassPointers.

Note that by default compressed class space + compressed heap (+ some overhead) cannot exceed 32GB. Although, AIUI, changing object alignment can bump that limit further.

Otherwise you should take a heapdump and analyze what's holding onto the loaded classes.

the8472
  • 40,999
  • 5
  • 70
  • 122
  • You can see in the jstat log that CCSU is 1048576.0 and CCSU is 11339.1. – user1236097 May 28 '15 at 16:20
  • 1
    might be fragmentation? I don't think class space gets compacted. Although such a huge factor is odd, even if it were fragmentation. Also, i think you're off by one column, still, a discrepancy remains. – the8472 May 28 '15 at 16:28
  • You can run this to see check if i wrong in columns: echo "S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT GCT\n 87296.0 87296.0 0.0 3151.4 523776.0 148284.4 3495936.0 574868.5 1395640.0 98066.3 1048576.0 11339.1 12165 636.851 223 116.957 753.808" | column -t – user1236097 May 28 '15 at 16:42
  • You're right. Very odd. Still, try those options and also enable heap dump on OOME as @Marged suggested. At the very least it should aid further diagnosis. – the8472 May 29 '15 at 17:19
1

You can fix this issue by using one of the two options below.

1. You can increase the limit of class space by using -XX:CompressedClassSpaceSize=n VM flag. Maximum limit is 4GB.

(OR)

2. You can completely disable Compressed class pointer feature by using -XX:-UseCompressedClassPointers VM flag.

Note: Please be aware that disabling this feature will increase heap space usage of your application.

From "Java Performance Optimization: Compressed OOPS" book

Each object has a pointer (_klass) to VM metadata class. This pointer is also compressed on 64-bit JVM. In JDK 8, permanent generation was removed and replaced by metaspace memory. This metaspace is dynamically growing memory and can reach native memory limit if the VM flags are not configured to control this metaspace memory. Prior Java 8, permanent memory is used and its size is fixed with VM flags. It can’t change its size at run time. When compared with heap memory space, this permanent memory space (non-heap) is very small and will usually be within 4GB limit to store class metadata and other metadata. So the virtual memory address range can be 0 to 4 GB if the zero-based virtual memory is assigned by operating system to this permanent memory. So compressing the memory addresses of klass objects using only 32-bits in this permanent memory location is not a problem.

But in Java 8, permanent memory is replaced with metaspace in order to support some performance improvements. This memory location can grow till it reaches the native memory limit if -XX: MaxMetaspaceSize VM is not set to control this growth. If it can grow till the native memory limit, then the klass objects might also be stored at any higher memory boundaries. This will be a problem to compress that memory location using 32-bit offset. So in order handle this problem and allocate a separate memory space for klass instances alone, Java introduced a new memory space called “compressed class space” in Java 8 version. The inclusion of compressed class space memory in Java 8 introduced “Java.lang.OutOfMemoryError: Compressed class space”. This OutOfMemoryError is triggered when the compressed class space memory reaches its maximum default capacity of 1 GB and there are no more room to store additional klass instances

In order to know more details about this OutOfMemoryError and about klass filed, you can read below books.

https://javaperformanceoptimization.com/

  1. Java Performance Optimization: Compressed OOPS
  2. Java Performance Optimization: How to avoid the 10 OutOfMemoryErrors
Nirmal
  • 89
  • 6