16

I need a way to do key-value lookups across (potentially) hundreds of GB of data. Ideally something based on a distributed hashtable, that works nicely with Java. It should be fault-tolerant, and open source.

The store should be persistent, but would ideally cache data in memory to speed things up.

It should be able to support concurrent reads and writes from multiple machines (reads will be 100X more common though). Basically the purpose is to do a quick initial lookup of user metadata for a web-service.

Can anyone recommend anything?

Jonas
  • 121,568
  • 97
  • 310
  • 388
sanity
  • 35,347
  • 40
  • 135
  • 226
  • What are you optimizing for? For example, read throughput (concurrent reads from multiple machines), fault tolerance in the face of machines becoming not available, low number of machines... Do you also need writes? – Alexander Oct 13 '08 at 15:38
  • Thanks, I've edited the question with this information. – sanity Oct 13 '08 at 15:41
  • How do you want your data distributed? Should all of the data be available to/on/from every node or not? In the first case the next question is "why the distributed lookup?". – Alexander Oct 13 '08 at 15:56

10 Answers10

12

You might want to check out Hazelcast. It is distributed/partitioned, super lite, easy and free.

java.util.Map map = Hazelcast.getMap ("mymap");
map.put ("key1", "value1");

Regards,

-talip

8

Open Chord is an implementation of the CHORD protocol in Java. It is a distributed hash table protocol that should fit your needs perfectly.

Nicholas Mancuso
  • 11,599
  • 6
  • 45
  • 47
2

Depending on the use case, Terracotta may be just what you need.

Alex Miller
  • 69,183
  • 25
  • 122
  • 167
1

You should probably specify if it needs to be persistent or not, in memory or not, etc. You could try: http://www.danga.com/memcached/

carson
  • 5,751
  • 3
  • 24
  • 25
0

Distributed hash tables include Tapestry, Chord, and Pastry. One of these should suit your needs.

0

OpenChord sounds promising; but i'd also consider BDB, or any other non-SQL hashtable, making it distributed can be dead-easy (if the number of storage nodes is (almost) constant, at least), just hash the key on the client to get the appropriate server.

Javier
  • 60,510
  • 8
  • 78
  • 126
0

Open Source Cache Solutions in Java

Oracle Coherence (used to be Tangosol)

JCache JSR

ykaganovich
  • 14,736
  • 8
  • 59
  • 96
0

Try distributed Map structure from Redisson, it based on Redis server. Using Redis cluster configuration you may split data across 1000 servers.

Usage example:

Redisson redisson = Redisson.create();

ConcurrentMap<String, SomeObject> map = redisson.getMap("anyMap");
map.put("123", new SomeObject());
map.putIfAbsent("323", new SomeObject());
map.remove("123");

...

redisson.shutdown();
Nikita Koksharov
  • 10,283
  • 1
  • 62
  • 71
0

nmdb sounds like its exactly what you need. Distributed, in memory cache, with a persistent on-disk storage. Current back-ends include qdbm, berkeley db, and (recently added after a quick email to the developer) tokyo cabinet. key/value size is limited though, but I believe that can be lifted if you don't need TICP support.

Phillip B Oldham
  • 18,807
  • 20
  • 94
  • 134
-1

DNS has the capability to do this, I don't know how large each one of your records is (8GB of tons of small data?), but it may work.

Ryan Stille
  • 1,364
  • 1
  • 13
  • 19