Issue with Hash Map Space and Performance

Question

I have to store more than 100 millions of key-values in my HashMultiMap (key can have multiple values). Can anybody help me which one is faster for both storing and searching:

1) Berkeley DB

2) Tokyo Cabinet

3) H2

4) EhCache

5) Or anyothers

Another point, is performance of those approximately identical to in-memory hash map ? A little bit guidance will be more helpful. Thanks.

NB: information about any one of these is also helpful.

@JeffFoster, more specifically http://stackoverflow.com/questions/9930347/multimap-space-issue-guava — Arpssss, Mar 30 '12 at 14:12
Are your ints randomly distributed over the 32bit space? I don't see why you couldn't store it all in memory (with a hand-written unboxed implementation of MultiMap). Alternatively you could try Redis. — Jeff Foster, Mar 30 '12 at 14:19
@JeffFoster, yes, distributed. I have to provide very limited memory to my JVM (2GB). If I want to store 100 millions ints, it will take more than 3 GB. — Arpssss, Mar 30 '12 at 14:27

score 3 · Accepted Answer · answered Mar 30 '12 at 14:33

3

I'd recommend Redis. It's more of a data structure store than the others (e.g. it supports maps and sets) and requires very little effort to get started with. It's incredibly simple to use and has a great Java API.

I'd start by looking at the functions here that allow you to associate a key with a group of values.

answered Mar 30 '12 at 14:33

Jeff Foster

43,770
11
86
103

Thanks a lot. From your experence, is performance of Redis is approximately identical to in-memory hash map ? – Arpssss Mar 30 '12 at 14:36
It'll depend on your circumstances. I've never had a problem with performance in Redis. Best advice would be to hack together a prototype and just see what happens. See also http://redis.io/topics/benchmarks – Jeff Foster Mar 30 '12 at 14:37

score 2 · Answer 2 · answered Mar 30 '12 at 19:54

I would have a look at leveldb, because it claims to be faster than Kyoto Cabinet. Other interesting libraries are Kyoto Cabinet, and JDBM3.

Those are low level libraries (without support for relational queries or SQL). Some offer transactional support, but I'm not sure if you need it. Still, in-memory hash maps are faster.

Issue with Hash Map Space and Performance

2 Answers2

Linked