4

Ehcache version

2.10.6

Once in a while, Ehcache servers are getting down because of the threads getting blocked. Please find below the details about the bucket configuration and thread details which are blocking each other.

There are multiple buckets: Bucket configuration

<cache name="bucket1"
           maxBytesLocalHeap="300m"
           maxBytesLocalDisk="15g"
           timeToIdleSeconds="240"
           timeToLiveSeconds="240"
           eternal="false"
           diskSpoolBufferSizeMB="30"
           memoryStoreEvictionPolicy="LFU"
           diskExpiryThreadIntervalSeconds="360"
           transactionalMode="off"
           diskPersistent="true">
    </cache>

Thread dump

http-nio-8080-exec-26
Stack Trace is:
java.lang.Thread.State: BLOCKED (on object monitor)
at net.sf.ehcache.store.cachingtier.OnHeapCachingTier$Fault.get(OnHeapCachingTier.java:332)
- **waiting to lock <0x0000000604407968>** (a net.sf.ehcache.store.cachingtier.OnHeapCachingTier$Fault)
at net.sf.ehcache.store.cachingtier.OnHeapCachingTier$Fault.access$200(OnHeapCachingTier.java:312)
at net.sf.ehcache.store.cachingtier.OnHeapCachingTier.getValue(OnHeapCachingTier.java:300)
at net.sf.ehcache.store.cachingtier.OnHeapCachingTier.access$000(OnHeapCachingTier.java:62)
at net.sf.ehcache.store.cachingtier.OnHeapCachingTier$1.evicted(OnHeapCachingTier.java:84)
at net.sf.ehcache.store.cachingtier.PooledBasedBackEnd$1.removed(PooledBasedBackEnd.java:195)
at net.sf.ehcache.util.concurrent.ConcurrentHashMap.internalReplace(ConcurrentHashMap.java:1346)
at net.sf.ehcache.util.concurrent.ConcurrentHashMap.removeAndNotify(ConcurrentHashMap.java:2647)
at net.sf.ehcache.store.cachingtier.PooledBasedBackEnd.remove(PooledBasedBackEnd.java:113)
at net.sf.ehcache.store.cachingtier.OnHeapCachingTier.remove(OnHeapCachingTier.java:207)
at net.sf.ehcache.store.CacheStore.put(CacheStore.java:134)
at net.sf.ehcache.Cache.putInternal(Cache.java:1616)
at net.sf.ehcache.Cache.put(Cache.java:1542)
at net.sf.ehcache.Cache.put(Cache.java:1507)......



http-nio-8080-exec-174
Stack Trace is:
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at net.sf.ehcache.util.concurrent.ConcurrentHashMap$Node.tryAwaitLock(ConcurrentHashMap.java:687)
- locked <0x00000005fa31d820> (a net.sf.ehcache.util.concurrent.ConcurrentHashMap$Node)
at net.sf.ehcache.util.concurrent.ConcurrentHashMap.internalReplace(ConcurrentHashMap.java:1311)
at net.sf.ehcache.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:2643)
at net.sf.ehcache.store.cachingtier.PooledBasedBackEnd.evict(PooledBasedBackEnd.java:159)
at net.sf.ehcache.store.cachingtier.PooledBasedBackEnd$PoolParticipant.evict(PooledBasedBackEnd.java:251)
at net.sf.ehcache.pool.impl.FromLargestCachePoolEvictor.freeSpace(FromLargestCachePoolEvictor.java:49)
at net.sf.ehcache.pool.impl.AtomicPoolAccessor.add(AtomicPoolAccessor.java:71)
at net.sf.ehcache.pool.impl.AbstractPoolAccessor.add(AbstractPoolAccessor.java:67)
at net.sf.ehcache.store.disk.Segment.put(Segment.java:427)
at net.sf.ehcache.store.disk.DiskStore.putFaulted(DiskStore.java:229)
at net.sf.ehcache.store.CacheStore$3.call(CacheStore.java:143)
at net.sf.ehcache.store.CacheStore$3.call(CacheStore.java:137)
at net.sf.ehcache.store.cachingtier.OnHeapCachingTier$Fault.get(OnHeapCachingTier.java:335)
**- locked <0x0000000604407968> (a net.sf.ehcache.store.cachingtier.OnHeapCachingTier$Fault)**
at net.sf.ehcache.store.cachingtier.OnHeapCachingTier$Fault.access$200(OnHeapCachingTier.java:312)
at net.sf.ehcache.store.cachingtier.OnHeapCachingTier.get(OnHeapCachingTier.java:176)
at net.sf.ehcache.store.CacheStore.put(CacheStore.java:137)
at net.sf.ehcache.Cache.putInternal(Cache.java:1616)
at net.sf.ehcache.Cache.put(Cache.java:1542)
at net.sf.ehcache.Cache.put(Cache.java:1507)
.......
Holy_diver
  • 377
  • 1
  • 15
  • Can you add more details as code or system ? do you use Spring/Hibernate/other? is there a load on your server or specific endpoint? – Ori Marko Jun 29 '20 at 13:52
  • No I am directly using ehcache through jar. Neither through hibernate nor through spring annotation – Holy_diver Jun 29 '20 at 14:05
  • can you show code or explain when you add to cache? – Ori Marko Jun 29 '20 at 14:13
  • It is a rest application meant just for caching, On PUT request, content is put into cache with the mentioned with simple vanilla syntax cache.put(Element). Load is on higher side sometimes reaching 50K requests per min (GET +PUT) . 16GB RAM, 100GB disk for caching – Holy_diver Jun 29 '20 at 14:17
  • @user7294900, 2 put threads are locking a resource, i am sensing something related to eviction. Can you work it out, as i am not sure – Holy_diver Jun 29 '20 at 17:19
  • Please share your code as well – Ajit Jul 02 '20 at 19:42
  • Hi @Ajit, It is rest application on tomcat. Simple cache.put and cache.get is used. APIS: PUT /cache/key payload and GET /cache/key .without any use of hibernate, spring cacheable – Holy_diver Jul 02 '20 at 20:05
  • 1
    not a real explanation but ... we had the same problem we fixed it by upgrading to ehcache 3.x (3.6 at the time) – benbenw Jul 06 '20 at 07:20
  • Thanks @benbenw, We are doing the same migrating to 3.8 . Your response serve as a good testimony for least. – Holy_diver Jul 06 '20 at 08:45
  • @benbenw, Can we discuss about migration on the chatroom: https://chat.stackoverflow.com/rooms/139/java – Holy_diver Jul 06 '20 at 08:51

1 Answers1

1

As stated in the comment, I don't have a real explanation but ...

We had the same problem with ehcache 2.10.5 : only heap cache and hibernate and standalone usage.

We fixed it by upgrading to ehcache 3.x (3.6 at the time).

We didn't investigate the issue on the 2.10 as it was an outdated version.

benbenw
  • 723
  • 7
  • 20