1

We're using

  1. OpenJDK 11
  2. Wildfly-17.0.1.Final
  3. Spring + Hibernate Application (Deployed as WAR in Wildfly)

With each deployment, the metaspace keeps on increasing and not getting garbage collected NOTE: We've set -XX:MaxMetaspaceSize parameter so that GC is triggered once the metaspace reaches the threshold

We are able to receive java.lang.OutOfMemoryError: Metaspace error upon reaching the MaxMetaspaceSize, but after that Full GC runs and doesn't clear any metaspace, below is the snippet of GC Log when it triggers Full GC due to threshold limit reached

[2023-03-21T05:54:15.335+0000] GC(1321) Pause Full (Metadata GC Threshold)
[2023-03-21T05:54:15.422+0000] GC(1317) Concurrent Sweep 159.546ms
[2023-03-21T05:54:15.422+0000] GC(1317) User=0.32s Sys=0.00s Real=0.16s
[2023-03-21T05:54:15.422+0000] GC(1321) Phase 1: Mark live objects
[2023-03-21T05:54:15.981+0000] GC(1321) Phase 1: Mark live objects 558.134ms
[2023-03-21T05:54:15.981+0000] GC(1321) Phase 2: Compute new object addresses
[2023-03-21T05:54:16.197+0000] GC(1321) Phase 2: Compute new object addresses 216.252ms
[2023-03-21T05:54:16.197+0000] GC(1321) Phase 3: Adjust pointers
[2023-03-21T05:54:16.832+0000] GC(1321) Phase 3: Adjust pointers 635.469ms
[2023-03-21T05:54:16.832+0000] GC(1321) Phase 4: Move objects
[2023-03-21T05:54:17.060+0000] GC(1321) Phase 4: Move objects 227.785ms
[2023-03-21T05:54:17.062+0000] GC(1321) Pause Full (Metadata GC Threshold) 443M->437M(907M) 1727.534ms
[2023-03-21T05:54:17.062+0000] GC(1320) ParNew: 5797K->0K(58944K)
[2023-03-21T05:54:17.062+0000] GC(1320) CMS: 452714K->448254K(869848K)
[2023-03-21T05:54:17.062+0000] GC(1320) Metaspace: 509715K->509715K(974848K)
[2023-03-21T05:54:17.063+0000] GC(1320) Pause Young (Metadata GC Threshold) 447M->437M(1232M) 1733.687ms
[2023-03-21T05:54:17.063+0000] GC(1320) User=1.67s Sys=0.05s Real=1.74s
[2023-03-21T05:54:17.063+0000] GC(1317) Old: 443369K->448254K(869848K)
[2023-03-21T05:54:17.063+0000] GC(1322) Pause Young (Metadata GC Threshold)
[2023-03-21T05:54:17.063+0000] GC(1322) Using 8 workers of 8 for evacuation
[2023-03-21T05:54:17.065+0000] GC(1323) Pause Full (Metadata GC Threshold)
[2023-03-21T05:54:17.066+0000] GC(1323) Phase 1: Mark live objects
[2023-03-21T05:54:17.631+0000] GC(1323) Phase 1: Mark live objects 565.499ms
[2023-03-21T05:54:17.631+0000] GC(1323) Phase 2: Compute new object addresses
[2023-03-21T05:54:17.899+0000] GC(1323) Phase 2: Compute new object addresses 267.410ms
[2023-03-21T05:54:17.899+0000] GC(1323) Phase 3: Adjust pointers
[2023-03-21T05:54:18.549+0000] GC(1323) Phase 3: Adjust pointers 649.816ms
[2023-03-21T05:54:18.549+0000] GC(1323) Phase 4: Move objects
[2023-03-21T05:54:18.631+0000] GC(1323) Phase 4: Move objects 82.086ms
[2023-03-21T05:54:18.633+0000] GC(1323) Pause Full (Metadata GC Threshold) 438M->437M(1232M) 1568.185ms
[2023-03-21T05:54:18.633+0000] GC(1322) ParNew: 10K->0K(391744K)
[2023-03-21T05:54:18.633+0000] GC(1322) CMS: 448254K->448254K(869848K)
[2023-03-21T05:54:18.633+0000] GC(1322) Metaspace: 509715K->509715K(974848K)
[2023-03-21T05:54:18.633+0000] GC(1322) Pause Young (Metadata GC Threshold) 437M->437M(1232M) 1570.109ms
[2023-03-21T05:54:18.633+0000] GC(1322) User=1.55s Sys=0.01s Real=1.57s
[2023-03-21T05:54:18.633+0000] GC(1324) Pause Young (Metadata GC Clear Soft References)
[2023-03-21T05:54:18.633+0000] GC(1324) Using 8 workers of 8 for evacuation
[2023-03-21T05:54:18.635+0000] GC(1325) Pause Full (Metadata GC Clear Soft References)
[2023-03-21T05:54:18.636+0000] GC(1325) Phase 1: Mark live objects
[2023-03-21T05:54:19.244+0000] GC(1325) Phase 1: Mark live objects 607.533ms
[2023-03-21T05:54:19.244+0000] GC(1325) Phase 2: Compute new object addresses
[2023-03-21T05:54:19.425+0000] GC(1325) Phase 2: Compute new object addresses 180.976ms
[2023-03-21T05:54:19.425+0000] GC(1325) Phase 3: Adjust pointers
[2023-03-21T05:54:19.931+0000] GC(1325) Phase 3: Adjust pointers 505.607ms
[2023-03-21T05:54:19.931+0000] GC(1325) Phase 4: Move objects
[2023-03-21T05:54:20.061+0000] GC(1325) Phase 4: Move objects 129.839ms
[2023-03-21T05:54:20.062+0000] GC(1325) Pause Full (Metadata GC Clear Soft References) 437M->385M(1232M) 1427.034ms
[2023-03-21T05:54:20.062+0000] GC(1324) ParNew: 0K->0K(391744K)
[2023-03-21T05:54:20.062+0000] GC(1324) CMS: 448254K->394889K(869848K)
[2023-03-21T05:54:20.062+0000] GC(1324) Metaspace: 509715K->509715K(974848K)
[2023-03-21T05:54:20.066+0000] GC(1324) Pause Young (Metadata GC Clear Soft References) 437M->385M(1232M) 1432.729ms
[2023-03-21T05:54:20.066+0000] GC(1324) User=1.43s Sys=0.01s Real=1.43s
[2023-03-21T05:54:20.066+0000] Metaspace (data) allocation failed for size 11
[2023-03-21T05:54:20.066+0000]
[2023-03-21T05:54:20.066+0000] Usage:
[2023-03-21T05:54:20.066+0000]   Non-class:    442.51 MB capacity,   435.47 MB ( 98%) used,     6.21 MB (  1%) free+waste,   856.81 KB ( <1%) overhead.
[2023-03-21T05:54:20.066+0000]       Class:     63.13 MB capacity,    59.73 MB ( 95%) used,     3.07 MB (  5%) free+waste,   342.06 KB ( <1%) overhead.
[2023-03-21T05:54:20.066+0000]        Both:    505.65 MB capacity,   495.19 MB ( 98%) used,     9.28 MB (  2%) free+waste,     1.17 MB ( <1%) overhead.
[2023-03-21T05:54:20.066+0000]
[2023-03-21T05:54:20.066+0000] Virtual space:
[2023-03-21T05:54:20.066+0000]   Non-class space:      448.00 MB reserved,     447.88 MB (>99%) committed
[2023-03-21T05:54:20.066+0000]       Class space:      504.00 MB reserved,      64.12 MB ( 13%) committed
[2023-03-21T05:54:20.066+0000]              Both:      952.00 MB reserved,     512.00 MB ( 54%) committed
[2023-03-21T05:54:20.066+0000]
[2023-03-21T05:54:20.066+0000] Chunk freelists:
[2023-03-21T05:54:20.066+0000]    Non-Class:  5.31 MB
[2023-03-21T05:54:20.066+0000]        Class:  1015.00 KB
[2023-03-21T05:54:20.066+0000]         Both:  6.30 MB
[2023-03-21T05:54:20.066+0000]
[2023-03-21T05:54:20.066+0000] MaxMetaspaceSize: 512.00 MB
[2023-03-21T05:54:20.066+0000] CompressedClassSpaceSize: 504.00 MB
[2023-03-21T05:54:20.066+0000]

Below are the configurations in standalone.conf

-Xms64m -Xmx2048m -XX:NativeMemoryTracking=detail -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m

We've taken Heap Dumps and further analysed using MAT and it seems org.jboss.modules.ModuleClassLoader is not getting cleared after subsequent deployments.

Same behaviour is observed even with undeployments.

Overview
Dominator Tree

We've tried below combinations, but the behaviour is same, and metaspace is not getting cleared

  1. openjdk11 + wildfly-17.0.1.Final

  2. openjdk11 + wildfly-18.0.1.Final

  3. openjdk16 + wildfly-17.0.1.Final

We are trying to find the root cause of this, does it have something related to internal implementation of JBoss Modules that it keeps the classes in memory even after undeployments or redeployments or are we missing something else here ?

Rahul Kumar
  • 91
  • 4
  • Is Spring installed as a module or included in your deployment? – James R. Perkins Mar 21 '23 at 14:53
  • Spring is not installed as a jboss module, it is part of the war artifact – Rahul Kumar Mar 22 '23 at 06:45
  • You'd want to check the path to GC so you can see what is holding it up from being collected. – James R. Perkins Mar 23 '23 at 14:55
  • I had checked that earlier and checked it again, and I found few paths leading to application code classes and after making few changes the retained heap size has come down to 36MB from 40MB per deployment, but still the major chunk is coming from internal `org.jboss.modules.ModuleClassLoader` – Rahul Kumar Mar 27 '23 at 09:55
  • Something must be holding a reference somewhere. It's possible, I don't know off the top of my head, there was a bug in WildFly 17 with references being held. The latest Jakarta EE 8 release of WildFly would be WildFly 26.1.3.Final. Not sure if it's possible to upgrade, but you may want to have a look. – James R. Perkins Mar 27 '23 at 15:05

0 Answers0