Ehcache not evicting cached entries

Question

Main question is does ehcahce 2.6.2 do ANY eviction of expired elements based on ehcache configuration; or do I have to evict programatcially?

After reading various parts of the documentation (data life and cache sizing) we set the current configuration to use TTI/TTL and a CacheManager level maxBytesLocalHeap=1024m. After setting this I would expect ehcache to only use up to 1Gb of heap space for caching, but from testing (and prodcution) we consistently see the JVM heap get full (2GB in our case)

Ehcache doesn't seem to honor the maxBytesLcoalHeap setting we set. Unless of course I (which is likely) dont' understand what this setting is actually doing.

Some notes from our admins:

During this test, the maxBytesLocalHeap value was 1024m which should have limited cache entries and references to 1GB total for the entire cache management pool. The 8 individual caches were configured to use a portion of the 1024m. The heap utilization climbed up in a steady fashion to the max heap size of 2GB. Once there, garbage collection took longer and response times degraded. The max heap size did not decrease after the test was over. In fact, it is roughly 30 hours after the end of the test, I cleared all cache entries via the CacheManagement page and allowed the JVM to settle over night, and the heap is still committed to 2GB. The use of the Old Generation space is flat with no room to spare. The use of the Young Generation space came down some after I cleared cache entries, but that was only about 240MB and the committed space did not reduce.

Thanks for your help and guidance!

Updated

ehcahce configuration

<?xml version="1.0" encoding="UTF-8"?>
<ehcache>

 <cache name="storeCache" 
   maxElementsInMemory="2500"
   memoryStoreEvictionPolicy="LFU"
   eternal="false"
   timeToLiveSeconds="86400"
   statistics="true"> 
   <persistence strategy="none" />
</cache>

<cache name="courseCache" 
   maxElementsInMemory="100000"
   memoryStoreEvictionPolicy="LFU"
   eternal="false"
   timeToLiveSeconds="43200"
   statistics="true"> 
   <persistence strategy="none" />
</cache>

<cache name="mtcIdCache" 
   maxElementsInMemory="500000"
   memoryStoreEvictionPolicy="LFU"
   eternal="false"
   timeToLiveSeconds="3600"
   statistics="true"> 
   <persistence strategy="none" />
</cache>

<cache name="catentryCache" 
   maxElementsInMemory="500000"
   memoryStoreEvictionPolicy="LFU"
   eternal="false"
   timeToLiveSeconds="3600"
   statistics="true"> 
   <persistence strategy="none" />
</cache>

<cache name="priceCache" 
   maxElementsInMemory="500000"
   memoryStoreEvictionPolicy="LFU"
   eternal="false"
   timeToLiveSeconds="300"
   statistics="true"> 
   <persistence strategy="none" />
</cache>

<cache name="inventoryCache" 
   maxElementsInMemory="500000"
   memoryStoreEvictionPolicy="LFU"
   eternal="false"
   timeToLiveSeconds="300"
   statistics="true"> 
   <persistence strategy="none" />
</cache>

<cache name="attributeCache" 
   maxElementsInMemory="500000"
   memoryStoreEvictionPolicy="LFU"
   eternal="false"
   timeToLiveSeconds="300"
   statistics="true"> 
   <persistence strategy="none" />
</cache>

<cache name="partnerCache" 
   maxElementsInMemory="10000"
   memoryStoreEvictionPolicy="LFU"
   eternal="false"
   timeToLiveSeconds="86400"
   statistics="true"> 
   <persistence strategy="none" />
</cache>
</ehcache>

** 2nd update **

This is what our admin says about the heap utilization

I have taken two memory snapshots for websvc01 JVM in relation to this endurance test. They are available in dynaTrace for Monday 09/09 @ 10:10:59 and Tuesday 09/10 @ 10:46:48. Both snapshots are Deep Memory Leak Analysis snapshots with capturing valued of string objects.

During this test, the maxBytesLocalHeap value was 1024m which should have limited cache entries and references to 1GB total for the entire cache management pool. The 8 individual caches were configured to use a portion of the 1024m. The heap utilization climbed up in a steady fashion to the max heap size of 2GB. Once there, garbage collection took longer and response times degraded. The max heap size did not decrease after the test was over. In fact, it is roughly 30 hours after the end of the test, I cleared all cache entries via the CacheManagement page and allowed the JVM to settle over night, and the heap is still committed to 2GB. The use of the Old Generation space is flat with no room to spare. The use of the Young Generation space came down some after I cleared cache entries, but that was only about 240MB and the committed space did not reduce.

Update:

Thanks for the responses I really appreciate it. I apologize for not updating this question for a while.

Using DynaTrace we were able to determine it is a problem with Spring Integration. Specifically, the SimpleMessageStore seems to be holding onto objects and filling up!

dynaTrace snapshot

Reading the documentation it seems I should be using a MessageReaper, but even after including this in the configuration I see the same issues. Should I be looking at the Capacity setting for the SimpleMessageStore instead?

Are you sure that the memory is consumed by ehcache ? if not take heap dump and explore it with eclipse memory analyser — jakcam, Sep 17 '13 at 15:37
I updated the question with the cache config. As it turns out not all the memory is being consumed by ehcahce. The cache is not being cleared as expected, not sure why. We have a LOT of java.util.concurrent.locks.ReentrantLock$NonfairSync, java.util.concurrent.ConcurrentHashMap$HashEntry, Java.util.concurrent.locks.ReentrantLock and java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject. in memory. [http://stackoverflow.com/questions/3959122/memory-fully-utilized-by-java-concurrenthashmap-under-tomcat](here). Basically, to shorten the tomcat session-timeout in web.xml. thoughts? — Carl, Sep 18 '13 at 18:54
Sorry, I messed up my link and explanation above. I was trying to say the only other explanation I could find regarding the excessive build up of those objects was [here](http://stackoverflow.com/questions/3959122/memory-fully-utilized-by-java-concurrenthashmap-under-tomcat). — Carl, Sep 18 '13 at 19:28
In your heap dump, can you see sizes of each cache instance? How cache elements are removed from your cache ? If i believe your admin said, your cache take only 240MB in memory. So two possibilities : 1/ your cache takes only 240MB and you have a memory leak, 2/ References from cache elements are held in your application and even if you clean the cache, those elements are not GC and your memory consumption stays high. — jakcam, Sep 21 '13 at 00:16

score 0 · Answer 1 · answered Sep 20 '13 at 15:04

0

In your ehcache configuration i do not see any memory limitation if you want specify memory limitation, you have to set the "maxBytesLocalHeap" attribute on the ehcache tag.

Ehcache documentation for more information : http://ehcache.org/documentation/configuration/cache-size#mixed-sizing-configurations.

answered Sep 20 '13 at 15:04

jakcam

1,789
2
12
14

Understood. We did try using that attribute. It just happens the above copy of ehache is from my local. We have tested using the attribute you mention. Heap still seems to build even using that attribute. – Carl Sep 20 '13 at 20:35

score 0 · Answer 2 · answered Dec 14 '13 at 13:53

0

Instead of 'maxElementsInMemory', use 'maxBytesLocalHeap' in your ehcache cache configuration to limit the heap usage.

answered Dec 14 '13 at 13:53

Sri

4,613
2
39
42

Ehcache not evicting cached entries

2 Answers2