Hazelcast: Predicate Performance Issue

Question

I am facing a performance issue in hazelcast while using the Predicate on the hazelcast map.

So I have a model class as shown below:

public class MyAccount {
   private String account;
   private Date startDate;
   private Date endDate;
   private MyEnum accountTypeEnum;

   // Overrides equals and hascodes using all the attributes
   // Has getters and setters for all the attributes
}

Then I create a hazelcast instance of type Hazelcast<MyAccount, String>. And in that instance I start saving the MyAccount object as key and associated string as it's value.

Point to note: I am saving these accounts in different maps (let say local, state, national and international)

Approx 180,000 objects of MyAccount is created and saved in the hazelcast, in different maps depending upon the account's geographical position. Apart from these, hazelcast stores another 50,000 string objects as keys and values in different maps (excluding the maps mentioned above)

Then I have a method which uses predicate filters on the attributes account, startDate and endDate to filter out accounts. Lets call this method as filter.

public static Predicate filter(String account, Date date) {        
    EntryObject entryObject = new PredicateBuilder().getEntryObject();
    PredicateBuilder accountPredicate = entryObject.key().get(Constants.ACCOUNT).equal(account);
    PredicateBuilder startDatePredicate = entryObject.key().get(Constants.START_DATE).isNull().or(entryObject.key().get(Constants.START_DATE).lessEqual(date));
    PredicateBuilder endDatePredicate = entryObject.key().get(Constants.END_DATE).isNull().or(entryObject.key().get(Constants.END_DATE).greaterThan(date));
    return accountPredicate.and(effectiveDatePredicate.and(endDatePredicate));
}

 private void addIndexesToHazelcast() { 
        Arrays.asList("LOCAL", "STATE", "NATIONAL", "INTERNATIONAL").forEach(location -> {
            IMap<Object, Object> map = hazelcastInstance.getMap(location);
            map.addIndex("__key." + "startDate", true);
            map.addIndex("__key." + "endDate", true);
            map.addIndex("__key." + "account", false);
        });
    }

Issue: For a particular map, say local, which holds around 80,000 objects, when I use the predicate to fetch the values from the map, it takes around 4 - 7 seconds which is unacceptable.

Predicate predicate = filter(account, date);
String value = hazelcast.getMap(mapKey).values(predicate); // This line takes 4-7 secs

I am surprised that the cache should take 4 - 7 seconds to fetch the value for one single account given that I have added index in the hazelcast maps for the same attributes. This is a massive performance blow.

Could anybody please let me know why is this happening ?

How big is your heap? You are probably paging memory to disk, which will make things very slow. — Chris Shain, Mar 22 '16 at 13:08
The total heap size as I see in Java Visual VM is around 1000 MB and the used heap size varies from 400 MB to 750 MB — RDM, Mar 22 '16 at 13:12
Probably not enough. Try bumping it up to 2GB and see if that improves things. — Chris Shain, Mar 22 '16 at 13:13
@ChrisShain: _You are probably paging memory to disk, which will make things very slow_. How do I set this thing right ? — RDM, Mar 22 '16 at 13:15
http://stackoverflow.com/questions/1565388/increase-heap-size-in-java — Chris Shain, Mar 22 '16 at 13:32
Please show the index definition as well and what HZ version do you use? — noctarius, Mar 22 '16 at 17:55
@noctarius: The hazelcast version is 3.6. It's kind of a continuation to my previous question which you had answered. http://stackoverflow.com/questions/36005130/hazelcast-issue-while-adding-index Before loading any data into hazelcast I am adding the index on the attributes `account, startDate and endDate` for each individual map i.e. `local, state, national and international`. I will update my question that particular piece of code shortly. — RDM, Mar 23 '16 at 02:34
@ChrisShain: I bumped up the max heap space to 5G but I don't see any kind of performance improvement in the process. — RDM, Mar 23 '16 at 05:04
Will you be able to share some of the code (and testdata) privately? That way we could have a look and possibly I could hand it over to engineering. But please see first if increasing the heap helps, also another GC might better the result (like ParNew + CMS). — noctarius, Mar 23 '16 at 05:34
@noctarius: I tried with 5 GB of heap space: `-Xmx5G` but there is no improvement in the performance. I can share the test code and test data within sometime. — RDM, Mar 23 '16 at 10:34
If you want to do it publicly please file a github issue otherwise send me a mail to chris @ hazelcast dot com :) — noctarius, Mar 23 '16 at 14:33
@noctarius: I have just now sent you a test code with mocked up data. Details are in the mail. Thanks. — RDM, Mar 23 '16 at 19:00
@pppavan: Yes the performance issue was resolved. You can refer this: https://stackoverflow.com/questions/36005130/hazelcast-issue-while-adding-index To add an index to a key, you should prefix the key with `__key#`. For example: `hazelcastInstance.getMap(location).addIndex(__key# + "startDate", true);` @noctarius: Please correct me if I am wrong. And I was wondering if you published a blog related to this issue. I knew that you had plans to do that. — RDM, Aug 08 '17 at 17:58
yeah performance improves with indexing.But it makes map to Eager load.Is there a way to map lazy and still add index @RahulDevMishra — nanpakal, Aug 09 '17 at 11:53
It would be a good idea to open up a new question on this topic — RDM, Aug 10 '17 at 20:46

Hazelcast: Predicate Performance Issue

0 Answers0