Java 8 Metaspace: OutOfMemoryError: Dealing with reflection Inflation

Question

There are lots of links and open Q&A s around web, still I'm missing a lot of information.

First things first

Issue:

java.lang.OutOfMemoryError: Metaspace

jvm:

java version:  "1.8.0_131"
vm:             Java HotSpot(TM) 64-Bit Server
vm args:       -Xms1024m -Xmx1024m -XX:MaxMetaspaceSize=128m

Frameworks:

spring, hibernate, wicket, jetty

Suspect 1:

Over the period of usage, the metaspace grows gradually and the following reflection classes are proportionally loaded into metaspace [observed by jmap -histo cron jobs]

sun.reflect.GeneratedConstructorAccessor1299
sun.reflect.GeneratedMethodAccessor6929
sun.reflect.GeneratedSerializationConstructorAccessor4220

Possible Solution:

a. As we are using libraries that are heavily dealing with reflection stuffs, we think 128m is quite not enough to hold all the generatedXX classes in metaspace. So we are planning to double the metaspace limit. -XX:MaxMetaspaceSize=256m

b. We are not thinking of setting the following

-D sun.reflect.noInflation

-D sun.reflect.inflationThreshold

Suspect 2:

The Full GC is running continuously even before it reaches/occupies the full configured metaspace(128m) and the application becomes unresponsive/slow/sometime OOM as the jvm only does the FGC.

[Full GC (Metadata GC Threshold) [PSYoungGen: 224K->0K(698368K)] [ParOldGen: 52910K->52933K(1398272K)] 53134K->52933K(2096640K), [Metaspace: 92733K->92733K(1163264K)], 0.1964143 secs] [Times: user=0.59 sys=0.00, real=0.19 secs]

.

Metaspace used 147414K, capacity 155731K, committed 159616K, reserved 1187840K
class space used 17242K, capacity 19252K, committed 20352K, reserved 1048576K

Possible Solution:

-XX:CompressedClassSpaceSize is not explicitly mentioned at the vm startup which may possibly cause over reserving the address space which leads to misleading committed sapce and hence the Full GC. So explicitly setting -XX:CompressedClassSpaceSize=256m will help vm in correct memory planning & reserving.

Questions:

Suspect 1: Did someone face similar issue and got any fix?
Suspect 2: Is setting -XX:CompressedClassSpaceSize really makes a difference in metaspace planning/reserving and affects GC? Any pointers?
Any other suspects? recommendations?

*-XX:CompressedClassSpaceSize is not explicitly mentioned at the vm startup which may possibly cause over reserving the address space which leads to misleading committed sapce* - reserved space does not count towards committed numbers / RSS / WSS. — the8472, Jul 06 '17 at 15:26
Just ***remove*** the `-XX:MaxMetaspaceSize=…` option. One of the greatest features of the meta space is that it can be resized. If you want to “help vm in correct memory planning & reserving”, don’t put arbitrary constraints on it. — Holger, Jul 07 '17 at 13:11
But that comes with the more risk when there is real class leak. — Sankarganesh Eswaran, Jul 07 '17 at 15:08

Sankarganesh Eswaran · Accepted Answer · 2017-08-11T12:08:51.820

After spending so much of time, it turns out there is no class leak and we are the victim of

"Reflection Inflation"

package sun.reflect;

/**
 The master factory for all reflective objects, both those in
 java.lang.reflect (Fields, Methods, Constructors) as well as their
 delegates (FieldAccessors, MethodAccessors, ConstructorAccessors).
**/

public class ReflectionFactory {


    // "Inflation" mechanism. Loading bytecodes to implement Method.invoke() and Constructor.newInstance() currently costs
    // 3-4x more than an invocation via native code for the first invocation (though subsequent invocations have been benchmarked
    // to be over 20x faster). Unfortunately this cost increases startup time for certain applications that use reflection
    // intensively (but only once per class) to bootstrap themselves. To avoid this penalty we reuse the existing JVM entry points
    // for the first few invocations of Methods and Constructors and then switch to the bytecode-based implementations.

I verified it personally, at every 16th time of refleciton generation this technique was used to improve the response time.

Finally we increase the metaspace and everything seems to work fine.

Java 8 Metaspace: OutOfMemoryError: Dealing with reflection Inflation

1 Answers1