Infinispace Cache: Inconsistent writes concurrent actions

Question

I have searched for solution to this issue everywhere but I could not find any hint.

My application is deployed on JBOSS EAP 6.4 and built using JDK 1.8. I have configured one local infinispan cache in standalone xml as:

<subsystem xmlns="urn:jboss:domain:infinispan:1.5">
    <cache-container name="test-cache" default-cache="test-data-cache" jndi-name="java:jboss/infinispan/test-cache" statistics-enabled="true">
        <transport lock-timeout="60000"/>
        <local-cache name="test-data-cache" statistics-enabled="true">
            <!-- <transaction locking="PESSIMISTIC"/> -->
            <locking isolation="READ_COMMITTED"/>
            <expiration lifespan="3600000"/>
        </local-cache>
    </cache-container>
</subsystem>

I put the data into the Cache as:

package com.comp.test;

import java.util.HashMap;
import java.util.Map;

import javax.annotation.Resource;
import javax.ejb.Stateless;
import javax.ejb.TransactionAttribute;
import javax.ejb.TransactionAttributeType;

import org.apache.commons.collections4.MapUtils;
import org.infinispan.Cache;

@Stateless
@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
public class CacheTest {

    @Resource(lookup = "java:jboss/infinispan/test-cache")
    private org.infinispan.manager.CacheContainer container;
    
    public void putData(String k, String v) {
        final Map<String, String> tmap = getDataCache().get(k);
        if (!containsData(k)) {
            final HashMap<String, String> rmap = new HashMap<>();
            rmap.put(k+"_"+v, v); 
            getDataCache().put(k, rmap);
        } else {
            getDataCache().get(k).put(k+"_"+v, v);
        }
    }
    
    public boolean containsData(String k) {
        return getDataCache().containsKey(k);
    }

    private Cache<String, Map<String, String>> getDataCache() {
        return container.getCache("test-data-cache");
    }
}

I have stateless beans which puts collection of data into the cache concurrently (@Asyncronous annotation). When I retrieve the data from the Cache once all concurrent operations are over, Cache always has the less number of values. If I put 20 values, only 16 / 17 values are present in the cache.

I tried to find out if I can put a lock on the key before I star putting data into the cache for that particular key. But I learnt that it is handled internally by the Infinispan. I found another similar question on SO. But this question is unanswered as well. Infinispan cache inconsistency for massive concurrent operations

Please let me know how to make sure that data put concurrently into the infinispan local cache is consistent at the end. If you need more information, please let me know.

score 2 · Accepted Answer · answered Apr 18 '21 at 09:51

2

You have two options that work with pessimistic locking:

There is a third option: use optimistic locking instead, and retry the transaction if another transaction modifies the key. But this would not work with @TransactionalAttribute, you would have to call TransactionManager.commit() yourself and catch the WriteSkewException.

answered Apr 18 '21 at 09:51

Dan Berindei

7,054
3
41
48

Thanks for the hint. I didnt have to start and stop transactions manually. I used "getDataCache().getAdvancedCache().lock(k)" and changed my cache configuration to make cache transactional and used PESSIMISTIC as a locking mode. I will post details in a separate answer. But I am accepting your answer as it gives a hint to lock the key. – user613114 Apr 18 '21 at 13:05
@user613114 there is one more thing I only noticed after you posted your answer: you are modifying the `HashMap` after inserting it in the cache, and that is not a good idea because a) readers will able to access the same instance while you're modifying it, and `HashMap` is not thread-safe (reads can even enter an infinite loop with the right timings) and b) if you ever decide to make the cache distributed, or store the data off-heap, the changes are not going to be applied correctly. – Dan Berindei Apr 18 '21 at 15:57
Thank you for your valuable feedbacks! 1) "reads can even enter an infinite loop with the right timings" -> how? What I read was infinispan allows async reads without any locks. 2) I do not see any possibility of making the cache distributed. I may make it replicated in case I implement the clustered mode. In that case there will be always a single node writing the data for the same key. Other nodes may read the data. Do you see any issue with that? – user613114 Apr 19 '21 at 05:19
@user613114 1) looks like my info was outdated: https://stackoverflow.com/questions/35534906/java-hashmap-getobject-infinite-loop 2) yes: if you modify the hashmap and don't call `cache.put(k, map)` then the map changes will not be seen by the other nodes – Dan Berindei Apr 19 '21 at 06:41

score 0 · Answer 2 · answered Apr 18 '21 at 13:11

I resolved the issue with the hint provided by @Dan Berindei.

I changed my cache configuration as:

<subsystem xmlns="urn:jboss:domain:infinispan:1.5">
    <cache-container name="test-cache" default-cache="test-data-cache" jndi-name="java:jboss/infinispan/test-cache" statistics-enabled="true">
        <transport lock-timeout="60000"/>
        <local-cache name="test-data-cache" start="EAGER" batching="false" statistics-enabled="true">
            <locking isolation="SERIALIZABLE" acquire-timeout="5000"/>
            <transaction mode="FULL_XA" locking="PESSIMISTIC"/>
            <expiration lifespan="3600000"/>
        </local-cache>
    </cache-container>
</subsystem>

Then I updated my code to lock key everytime I invoke put on the Cache:

package com.comp.test;

import java.util.HashMap;
import java.util.Map;

import javax.annotation.Resource;
import javax.ejb.Stateless;
import javax.ejb.TransactionAttribute;
import javax.ejb.TransactionAttributeType;

import org.apache.commons.collections4.MapUtils;
import org.infinispan.Cache;

@Stateless
@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
public class CacheTest {

    @Resource(lookup = "java:jboss/infinispan/test-cache")
    private org.infinispan.manager.CacheContainer container;
    
    public void putData(String k, String v) {
        final Map<String, String> tmap = getDataCache().get(k);
        if (!containsData(k)) {
            final HashMap<String, String> rmap = new HashMap<>();
            rmap.put(k+"_"+v, v); 
            getDataCache().put(k, rmap);
        } else {
                 AdvancedCache<String, Map<String, String>> cache = getDataCache().getAdvancedCache();
                cache.lock(k);
            getDataCache().get(k).put(k+"_"+v, v);
        }
    }
    
    public boolean containsData(String k) {
        return getDataCache().containsKey(k);
    }

    private Cache<String, Map<String, String>> getDataCache() {
        return container.getCache("test-data-cache");
    }
}

Infinispace Cache: Inconsistent writes concurrent actions

2 Answers2