10

I troubleshoot a weird memory leak. It's specific to Java8, not happening on 7u79.

I don't have access to Java code. I know exactly which user action causes a leak, I know the leak is about the classes (not the heap) and the offending classes are easy to spot with +TraceClassLoading +TraceClassUnloading:

[Loaded com.mastercard.mcwallet.sdk.xml.allservices.ShoppingCartRequest$JaxbAccessorF_oAuthToken from __JVM_DefineClass__]
[Loaded com.mastercard.mcwallet.sdk.... thousand similar classes per one user action... ]

These classes seem to increase the class counter output by jstat -class:

Loaded  Bytes  Unloaded  Bytes     Time
 14045 26138.8        0     0.0     110.00   << buggy user action
 14675 26754.6        0     0.0     110.05
 15300 27364.9        0     0.0     110.10
 15304 27370.9        0     0.0     110.11
 15304 27370.9        0     0.0     110.11
 15304 27370.9        0     0.0     110.11
 15306 27374.0        0     0.0     110.11
 15306 27374.0        0     0.0     110.11
 15306 27374.0        0     0.0     110.11
 15306 27374.0        0     0.0     110.11   << buggy user action
 15930 27982.2        0     0.0     110.18
 16553 28589.3        0     0.0     110.23
 16553 28589.3        0     0.0     110.23

The thing is that these classes are never garbage-collected from metaspace, never [Unloaded] and they don't show in jmap -clstats. The command reports a lower number of classes, the number doesn't increase, there are no suspicious class loaders:

class_loader    classes bytes   parent_loader   alive?  type

<bootstrap>     2574    4493256   null          live    <internal>
0x0000000087d016d0      1       1471    0x000000008237f088      dead    sun/reflect/DelegatingClassLoader@0x00007ff4135d02d0
... some lines omitted ...
0x000000008237f088      6505    12228227        0x0000000080383938      dead    org/apache/catalina/loader/WebappClassLoader@0x00007ff411546ad0
... some lines omitted ...
total = 600     14002   25351427            N/A         alive=1, dead=599           N/A

Does this ring any bell or brings any tips I could pass to programmers? They say they are unable to locate the leak. Can I stop this leak just by fiddling with JVM options?

kubanczyk
  • 5,184
  • 1
  • 41
  • 52
  • The problem only with `MetaSpace`? what about `heap`? and can you also provide the `Java` process memory usage statistic? – Sergey Bespalov Sep 30 '16 at 03:12
  • Only Metaspace, heap is stable and not full. What statistic exactly? – kubanczyk Sep 30 '16 at 07:07
  • I mean how fast Java process memory usage grows? – Sergey Bespalov Sep 30 '16 at 07:49
  • Each "buggy user action" allocates about 2 MB from OS (`pmap` tool even shows a separate 2 MB segment). – kubanczyk Sep 30 '16 at 07:53
  • have you tried to use different GC algorithms, for example G1 `-XX:+UseG1GC`? Or you can try to set `-XX:MaxMetaspaceSize=` to expectedly define the size of `Metaspace`. – Sergey Bespalov Oct 03 '16 at 03:36
  • Yup, tried them all, no difference at all. These classes seem simply not eligible for GC in Java8, but somehow can disappear on Java7. With MaxMetaspaceSize the app simply dies as soon as it fills up. – kubanczyk Oct 03 '16 at 06:07
  • You can try to use `Eclipse Memory Analyzer` ( [MAT](https://eclipse.org/mat/) ) to inspect your heap dump. If buggy classes unavaliable for unloading then it must be cached somewhere in application, and MAT can show you where it is. – Sergey Bespalov Oct 03 '16 at 08:32

2 Answers2

6

If buggy classes unavailable for unloading then it must be cached somewhere in application, and as I understand your goal is to find where it is and provide this information for developers.

You can try to use Eclipse Memory Analyzer (MAT) to inspect your heap dump, and it can show problem areas, classes and instances which has links to the buggy classes.

Sergey Bespalov
  • 1,746
  • 1
  • 12
  • 29
4

I presume JaxB is loading classes into your memory, which are not being garbage collected. Metaspace will not empty out if the Classes are not unloaded.

Check for JaxB memory issue for Java 8. I think you can get rid of this exception by swapping the dependency api version or by toggling on an extra parameter:

-Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true
Community
  • 1
  • 1
Abinash
  • 276
  • 1
  • 6