23

This is a memory stack (serves as a cache) that consist of nothing but a static ConcurrentHashMap (CHM).

All incoming HTTP request data are store in this ConcurrentHashMap. And there is a asynch scheduler process that takes the data from the same ConcurrentHashMap and remove the key.value after storing them into the Database.

This system runs fine and smooth but just discover under following criteria, the memory was fully utilized (2.5GB) and all CPU time was taken to perform GC:

-concurrent http hit of 1000/s

-maintain the same concurrent hit for a period of 15 minutes

The asynch process log the remaining size of the CHM everytime it writes to database. The CHM.size() maintain at around Min:300 to Max:3500

I thought there is a Memory Leak on this application. so i used Eclipse MAT to look at the Heap Dump. After running the Suspect Report, i got these comments from MAT:

One instance of "org.apache.catalina.session.StandardManager" loaded by "org.apache.catalina.loader.StandardClassLoader @ 0x853f0280" occupies 2,135,429,456 (94.76%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "".

3,646,166 instances of java.util.concurrent.ConcurrentHashMap$Segment retain >= 2,135,429,456 bytes.

and

Length    # Objects      Shallow Heap      Retained Heap 
0         3,646,166      482,015,968       >= 2,135,429,456

The length 0 above i translate it as empty length record inside the CHM (each time i call CHM.remove() method). It is consistent to the number of record inside the database, 3,646,166 records was inside the database when this dump was created

The strange scenario is: if i pause the stress test, the utilization in Heap Memory will gradually release down to 25MB.This takes about 30-45 minutes. i have re-simulate this application and the curves looks similar to the VisualVM Graph below: alt text

Heres the questions:

1) Does this looks like a Memory Leak?

2) Each remove call remove(Object key, Object value) to remove a <key:value> from CHM, does that removed object get GC?

3) Is this something to do with the GC settings? i have added the following GC parameters but without help: 

-XX:+UseParallelGC

-XX:+UseParallelOldGC

-XX:GCTimeRatio=19

-XX:+PrintGCTimeStamps

-XX:ParallelGCThreads=6

-verbose:gc

4) Any idea to resolve this is very much appreciated! :)

NEW 5) Could it be possible because all my reference are hard reference? My understanding is as long as the HTTP session is ended, all those variables that is not static are now available for GC.

NEW Note I tried replace the CHM with ehcache 2.2.0, but i get the same OutOfMemoryException problem. i suppose ehcache is also using ConcurrentHashMap.

Server Spec:

-Xeon Quad core, 8 threads.

-4GB Memory

-Windows 2008 R2

-Tomcat 6.0.29

Reusable
  • 1,888
  • 7
  • 27
  • 46
  • How hard would it be to replace the hash map with an instance of EhCache? These libraries are optimized for this kind of tasks. – Boris Pavlović Oct 18 '10 at 12:43
  • At the moment we try not to change that much to the existing code because we are yet to analyse the impact. EhCache was part of the consideration initially but somehow was not choosen as the implementation choice. – Reusable Oct 19 '10 at 08:08

5 Answers5

11

This problem has bug me for a bad 7 days! And finally i found out the real problem! Below are the tasks on what i have tried but failed to solve the OutOfMemory Exception:

-change from using concurrenthashmap to ehcache. (turns out ehcache is also using ConcurrentHashMap)

-change all the hard reference to Soft Reference

-Override the AbstractMap along side with concurrnetHashMap as per suggest by Dr. Heinz M. Kabutz

The million dollar question is really "why 30-45 minutes later, memory starting to release back to the heap pool?"

The actual root cause was because there is something else still holding the actual variable session, and the culprit is the http session within tomcat is still active! Hence, even though the http session was completed, but if the timeout setting is 30 minutes, tomcat will hold the session information for 30 minutes before JVM can GC those. Problem solve immediately after changing the timeout setting to 1 minute as testing.

$tomcat_folder\conf\web.xml

<session-config>
    <session-timeout>1</session-timeout>
</session-config>

Hope this will help anyone out there with similar problem.

Reusable
  • 1,888
  • 7
  • 27
  • 46
  • I wonder if changing to Persistent session manager (using File store) solved your problem while keeping the session-timeout ? – Hendy Irawan Sep 14 '13 at 08:26
  • It might. if only i have time to retest this – Reusable Sep 18 '13 at 03:57
  • Did you figure out why the session had not been closed? What variables were constantly referencing to the objects from a session? – Zimas Mar 26 '18 at 10:34
  • It's totally normal, an HttpSession never gets closed by the client, the server keeps it around in case further requests come in until the timeout expires. As others have remarked this is fundamentally a flaw in his test design, in which new sessions keep getting created. – volkerk Aug 31 '21 at 13:24
10

I think you're using too much session data that won't fit at once in memory. Try this one:

  1. Edit bin/setenv.sh or wherever the JVM args are set on your Tomcat launcher :

    Append -Dorg.apache.catalina.session.StandardSession.ACTIVITY_CHECK=true

    e.g. 

    # Default Java options
if [ -z "$JAVA_OPTS" ]; then
        JAVA_OPTS="-server -Djava.awt.headless=true -XX:MaxPermSize=384m -Xmx1024m -Dorg.apache.catalina.session.StandardSession.ACTIVITY_CHECK=true"
fi

  2. Edit conf/context.xml, before </Context> add this:

    <Manager className="org.apache.catalina.session.PersistentManager"
        maxIdleBackup="60" maxIdleSwap="300">
    <Store className="org.apache.catalina.session.FileStore"/>
</Manager>

Restart Tomcat and your problem should be gone, since it'll store your sessions using the filesystem instead.

In my view setting session-timeout = 1 is a workaround that masks the root of problem, and is unusable in most apps where you actually need a big enough session-timeout. Our (Bippo's) apps usually have a session-timeout of 2880 minutes i.e. 2 days.

Reference: Tomcat 7.0 Session Manager Configuration

Hendy Irawan
  • 20,498
  • 11
  • 103
  • 114
  • Hi, how can I achieve this when using the built-in spring-boot server? What configuration should I change? – malkoto1 Nov 25 '18 at 09:25
3

1) Does this looks like a Memory Leak?

Yes, if the application keeps on putting objects in the map and never removes them, then that could very well be a memory leak.

2) Each remove call remove(Object key, Object value) to remove a from CHM, does that removed object get GC?

Objects can only be garbage collected if there is no live (running) thread that has a reference to them. The map is only one place where there's a reference to the object. There could still be other places that have references to the same object. But keeping the object in the map will prevent it from being garbage collected.

3) Is this something to do with the GC settings?

No; if an object is referenced, it cannot be garbage collected; it doesn't matter how you tweak the garbage collector.

Jesper
  • 202,709
  • 46
  • 318
  • 350
  • 2nd and 3 question i have to agree with you fully. But as for 1st question, If it is a memory leak, any possibility that after 30-45 minutes, the Java Heap Memory utilization goes back to its initialization state? – Reusable Oct 18 '10 at 13:04
  • @Reusable if that is what is occurring then it shouldn't be labeled a true "memory leak". However if you are finding that memory is not reclaimed after you think it ought to be (because you are no longer referencing the data) then this suggests some logic in your Map-usage is flawed. – matt b Oct 18 '10 at 13:28
  • @matt b i am thinking along the same line with you but is purely my rough guess, that there is some other object(s) is holding the reference of either the key/value that is removed. Without any success and still checking the codes. – Reusable Oct 19 '10 at 08:33
  • Are you using Eclipse? If so there is an excellent plugin for analyzing heap dumps to find memory leaks: http://www.eclipse.org/mat/. [See here](http://wiki.eclipse.org/index.php/MemoryAnalyzer#Getting_a_Heap_Dump) for info on how to generate a heap dump. – matt b Oct 19 '10 at 12:37
  • @matt b I am already using Eclipse MAT :) I have edited the main question above with the summary and suspect report generated by MAT. However, i do not see any classes still keeping any reference to its CHM key/value that prevent it from GC :( – Reusable Oct 20 '10 at 00:43
  • Uh oh.. I just found this question ask from another thread in Stackoverflow: http://stackoverflow.com/questions/2811537/is-java-hashmap-clear-and-remove-memory-effective Is it true that HashMap does not reduce the size even a .remove() is call? i even try to set the value to "" empty string before calling .remove() but the memory still built up like a saw and eventually outofmemoryexception. – Reusable Oct 20 '10 at 03:17
1

Of course, it is too late to answer, but just for other people who will find this question by search. It might be useful.

These 2 links are very useful
https://issues.apache.org/bugzilla/show_bug.cgi?id=50685
http://wiki.apache.org/tomcat/OutOfMemory

Briefly, in most cases it is a wrong test or test software. When some custom software open URL, if this software cannot manage http session tomcat creates new session for each request. For example it is possible to check it with simple code, which can be added to JSP.

System.out.println("session id: " + session.getId());
System.out.println("session obj: " + session);
System.out.println("session getCreationTime: " + (new Date(session.getCreationTime())).toString());
System.out.println("session.getValueNames().length: " + session.getValueNames().length);

If session ID will be the same for one user from load test point of view, it is fine, if each request generates new session ID, that means testing software does not manage the sessions very well and test result does not represent load from real users.

For some application session.getValueNames().length also important, because For example, when normal user works it remain the same, but when load testing software do the same, it is grows. It also means, that load testing software does not represent real workload very well. In my case session.getValueNames().length for normal user was about 100, but qwith load testing software after 10 minutes is was about 500 and finally system crashes with the same OutOfMemory error and MAT shows the same:

org.apache.catalina.loader.StandardClassLoader @ 0x853f0280" occupies 2,135,429,456 (94.76%) bytes.

Zlelik
  • 559
  • 1
  • 7
  • 17
0

If you get this exception and are using spring boot version 1.4.4 RELEASE or lower, set value of property "server.session-timeout" in minutes, rather than what they suggest (seconds), so that the sessions on the heap will be cleaned in time. Or you can use a bean of EmbeddedServletContainerCustomizer but the provided value will be set in minutes.

example (session timeout in 10 minutes): server.session-timeout=10 (set in properties file) container.setSessionTimeout(10, TimeUnit.SECONDS); (set in EmbeddedServletContainerCustomizer)

user1819111
  • 381
  • 3
  • 6