CQEngine Optimize for Small DataSets

Question

I have an application which needs to apply flexible queries against millions of smaller collections ranging in size from 10 to 10000 items per collection. CQEngine is working great in terms of providing the flexibility to query these collections but is much slower than the previous more rigid implementation which worked by pre computing aggregates over certain attributes of the items in the collection. Problem with that method is it was not flexible enough to easily handle the addition of new attributes.

My question is for processing millions of smaller collections is there anything I can do to tune CQEngine to make it faster.

1. Should I add indexes, or maybe only add indexes if the collection is over some size.
2. I am currently using a Navigable on the record timestamp and HashIndex on the other attributes which are things like category or tag.

Any ideas would be greatly appreciated!

Answer 1

Obviously it depends on your queries, but yes look at your indexes you're using. Navigable sounds ok for timestamp, although you could do a compound on the parts of the timestamp (year then month then day .. etc)

Determine the cardinality of the different attributes of your objects and prioritize the indexing by that -- using hashes for high-caridnal items and trees for the lower cardinal attributes.