9

Issue description

I noticed that each deployment of one of our Java8 application on Wildfly 8.2.1 uses approximately 30-40 MBs from Metaspace memory pool. That is fine, but the thing is that once I am redeploying the same app the Metaspace memory usage is being increased by the same 30-40 MBs whereas the old already allocated memory is not being released.

I would not even notice it but the thing is that we have ~20 applications and from time to time I need to redeploy up to 10 of them at the same time. That in turn results in a scary picture.

enter image description here Basically what is shown are the 2 redeployments of ~10 applications.

I'm not sure why GC cannot release the memory allocated to the old classes. This server has in total 16GB physical memory so I can redeploy all apps up to 20-40 times and that is it. App server will reach the limit and will stop respond to any commands.

So I would be very thankful if someone could help me to understand what the actual issue could be:

  1. Is it a Java8 issue? (jdk 1.8.0_40-b26)
  2. Is it a Wildfly 8.2.1 issue?
  3. Or is the answer as simple as possible that the cause is my code base? If so, then could you please guide me what could be the actual reason?

Some more details related to my code base

1) Together with Wildfly I use 2 standalone HornetQ servers, each application uses ~5 channels with at least 5 concurrent consumers on each. Which in turn results in at least 25 threads for each app and at least 25*20 = 500 threads in total.

2) For all low-level JMS operations I use Spring JMS.

blubb
  • 9,510
  • 3
  • 40
  • 82
Andrei
  • 303
  • 3
  • 7

2 Answers2

4

To empirically determine where and if you have a leak (you may not have a leak - it could just be legitimate classes getting loaded during deployments for legitimate reasons) you might try taking a heap dump at the right time (i.e. before or after the increase in the metaspace occurs). Then, take another heap dump. Diff the two heap dumps using a tool like MAT or Yourkit. The diff between the two will tell you which classes are being loaded.

Leaks in the metaspace are pretty rare because you only have so many classes to load but it could be something pretty exotic at play as well.

Let me know what you find and happy hunting! These are fun. :-)

Community
  • 1
  • 1
entpnerd
  • 10,049
  • 8
  • 47
  • 68
  • I analyzed both heap dumps (before and after redeploying ~5 artifcats) and noticed something suspecious even before calculating the difference: 1. before http://snag.gy/JGZdf.jpg 2. after http://snag.gy/VqFiN.jpg And here is the difference: http://snag.gy/xfDeV.jpg I see that char[] array is growing a lot. (seems like content of String) Here is the content of org.jboss...JavaZipFileSystem class loader: http://snag.gy/bbYq7.jpg So I have an impression that WildFly simply loads all the artifacts into the memory and doesn't release them even after redeployment. Could that be true and is it fine? – Andrei Oct 20 '15 at 15:27
  • 1
    Cool! Here are a few suggestions: 1. You want a few dumps in between deployments to find a pattern of monotonically increasing mem usage, filtering out random objects. 2. Try to find data within the objects you suspect are causing the leak (i.e. values in nodes); that can lead your forensics. 3. Be sure to investigate the meta space where you suspect the leak is, and not the heap. 4. Yes, char[] types and other primitives fluctuate a lot. It's easier to find the leak at higher levels. 5. Yes, WildFly could easily be loading classes and not releasing them. Class unloading is optional for a JVM. – entpnerd Oct 21 '15 at 06:08
  • Did you find any solution to this ? I am having same issue – Deibys Feb 29 '16 at 22:42
1

WildFly 10.0.0.Final "java.lang.OutOfMemoryError: Metaspace" occurs and will be fixed. Refer to follow Agile Board of Wildfly

https://issues.jboss.org/browse/WFLY-6173

Kevin
  • 42
  • 3
  • Welcome to Stack Overflow! A link to a potential solution is always welcome, but please [add context around the link](http://meta.stackoverflow.com/a/8259/169503) so your fellow users will have some idea what it is and why it’s there. Always quote the most relevant part of an important link, in case the target site is unreachable or goes permanently offline. Take into account that being barely more than a link to an external site is a possible reason as to [Why and how are some answers deleted?](http://stackoverflow.com/help/deleted-answers). – ddb Jul 27 '16 at 07:44
  • 1
    The context is there. It is a bug in the application and will be fixed. Given that this error is common and this page one of the top google hits for the problem, by deleting this answer you'd be causing more damage to the community than good. – tggm Feb 03 '17 at 10:08