java server with jetty memory leak issue

Question

I´m running this application with a Jetty server embedded. The application is slowly consuming more memory dispite the fact that the heap size does not change at all.

this is the command to start the application :

java -server -Xms1G -Xmx1G -Dfile.encoding=UTF-8

This is the chart from new relic with the RAM usage increase it started using 1140MB and after 8 hours finished with 1290MB

This is the jvisualvm chart of the heap.

Any suggestions ?

[EDIT1] add dump https://www.dropbox.com/s/1gt1i9dhjtjauf0/gameserver-20160226-2107.zip?dl=0

[EDIT2] Here are some notes from my current investigation

= How Monitor memory outside the heap add parameters

-XX:NativeMemoryTracking=detail -XX:+UnlockDiagnosticVMOptions -XX:+PrintNMTStatistics

run command jcmd 3322 VM.native_memory summary @see https://devcenter.heroku.com/articles/java-memory-issues

= Checking the default thread stack size java -XX:+PrintFlagsFinal -version | grep ThreadStackSize

intx ThreadStackSize = 1024
How to reclaim the memory used by a Java thread stack?

= Understand the max memory Max memory = [-Xmx] + [-XX:MaxPermSize] + number_of_threads * [-Xss] @see https://plumbr.eu/blog/memory-leaks/why-does-my-java-process-consume-more-memory-than-xmx

= Actions Reduce the size of the thread stacks. (In theory, you can go as low as 64K ...)

-XX:ThreadStackSize=256

Reduce the Minimum percentage of heap free after GC to avoid expansion.

-XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=10

Now will continue monitoring ...

You might have forgotten to close a `Closeable` such as a `Connection`, a `Transaction`, `FileOutputStream` etc. It can also come from the usage of a native library. — Arnaud Denoyelle, Feb 26 '16 at 16:33
Thanks @ArnaudDenoyelle, but I believe that would show at the heap usage chart, and the heap is allways bellow 400MB (sad) — hamilton.lima, Feb 26 '16 at 16:38
It is impossible to say if you have a memory leak at all from the information provided. — Mark Rotteveel, Feb 26 '16 at 17:45
what else should I provide ? pleae enlightme @MarkRotteveel !! :) — hamilton.lima, Feb 26 '16 at 18:24
@hamilton.lima post an HPROF heap dump somewhere, then we can have a look at it — Ingo Kegel, Feb 26 '16 at 21:00
Impossible to say, all you have posted is an indication that your OS is claiming more memory, this could also be for cache that will be released if necessary. — Mark Rotteveel, Feb 26 '16 at 22:34
@IngoKegel here is the HPROF https://www.dropbox.com/s/1gt1i9dhjtjauf0/gameserver-20160226-2107.zip?dl=0 — hamilton.lima, Feb 27 '16 at 00:28
sadly @MarkRotteveel whatever the SO is claiming is not been released — hamilton.lima, Feb 27 '16 at 00:29
investigating http://stackoverflow.com/questions/2756798/java-native-memory-usage thread stack seems to be the reason, as the heap does not change — hamilton.lima, Feb 27 '16 at 01:53

score 1 · Accepted Answer · answered Feb 27 '16 at 23:21

After lots of different types of investigations, I came back to the Eclipse Memory analyzer and tried once more ...

But this time I decided to trust the leak report that was saying :

One instance of "org.hibernate.internal.SessionFactoryImpl" loaded by "sun.misc.Launcher$AppClassLoader @ 0xc001d4f8" occupies 2.956.808 (21,05%) bytes. The memory is accumulated in one instance of "org.hibernate.internal.SessionFactoryImpl" loaded by "sun.misc.Launcher$AppClassLoader @ 0xc001d4f8".

Then I shift my investigation the my DAO implementation, expecting that I forgot to close one or other EntityManager call. That wasn´t the case, all of then were with a nice close() method call after use.

Then I realize that the problem could be on hibernate itself as the problematic object were a SessionFactoryImpl, so I changed my DAO implementation to clear Hibernate first level cache every time I create an Entity Manager, as I wasn´t able to find a way to disable it.

End of the day result = IT WORKED !! :) the memory changes a litle and come back after some minutes, but don´t grow in a crazy way anymore (1000MB in 24hours).

Here is the code I changed, hope it helps someone.

public EntityManager getEntityManager(){
    if( emf == null ){
        if (parameters == null) {
            emf = Persistence.createEntityManagerFactory(persistenceUnitName);
        } else {
            emf = Persistence.createEntityManagerFactory(persistenceUnitName, parameters);
        }
    } else {
        emf.getCache().evictAll();
    }

    EntityManager em = emf.createEntityManager();
    return em;
}

The key is here : emf.getCache().evictAll();

java server with jetty memory leak issue

1 Answers1