Are they any decent on-disk implementations of Java's Map?

Question

I'm looking for an on-disk implementation of java.util.Map. Nothing too fancy, just something that I can point at a directory or file and have it store its contents there, in some way it chooses. Does anyone know of such a thing?

aioobe · Accepted Answer · 2016-01-09T21:39:47.560

10

You could have a look at the Disk-Backed-map project.

A library that implements a disk backed map in Java

A small library that provide a disk backed map implementation for storing large number of key value pairs. The map implementations (HashMap, HashTable) max out around 3-4Million keys/GB of memory for very simple key/value pairs and in most cases the limit is much lower. DiskBacked map on the other hand can store betweeen 16Million (64bit JVM) to 20Million(32bit JVM) keys/GB, regardless the size of the key/value pairs.

edited Jan 09 '16 at 21:39

answered Jan 27 '11 at 11:22

aioobe

413,195
112
811
826

This is an accurate quotation from their website, but it must be wrong. It makes no sense for a 64-bit JVM to have smaller collections than a 32-bit JVM. – chiastic-security Sep 09 '15 at 09:28
I'd guess the lower keys/GB is due to 64-bit addresses being twice the size of 32-bit addresses. – Robert Sharp Oct 21 '18 at 03:18

score 6 · Answer 2 · answered Feb 19 '14 at 16:28

6

MapDB (mapdb.org) does exactly what you are looking for. Besides disk backed TreeMap and HashMap it gives you other collection types.

It's maps are also thread-safe and have really good performance.

See Features

answered Feb 19 '14 at 16:28

Andrejs

26,885
12
107
96

1

It looks nice on paper, but in my experience it's sadly utterly slow and not crash safe at all (even with `transactionEnable`). – dagnelies Jul 10 '17 at 09:47

Manuel Salvadores · Answer 3 · 2011-02-18T13:51:09.213

If you are looking for key-object based structures to persist data then NoSQL databases are a very good choice. You'll find that some of them such MongoDB or Redis scale and perform for big datasets and apart from hash based look ups they provide interesting query and transactional features.

In essence these types of systems are a Map implementation. And it shouldn't be too complicated to implement your own adapter that implements java.util.Map to bridge them.

score 2 · Answer 4 · answered Mar 19 '17 at 00:20

2

Chronicle Map is a modern and the fastest solution to this problem. It implements ConcurrentMap interface and persists the data to disk (under the hood, it is done by mapping Chronicle Map's memory to a file).

answered Mar 19 '17 at 00:20

leventov

14,760
11
69
98

score 1 · Answer 5 · answered Jan 27 '11 at 11:24

1

You could use a simple EHCache implementation? The nice thing about EHCache being that it can be very simple to implement :-)

I take it you've ruled out serialising / deserialising an actual Map instance?

answered Jan 27 '11 at 11:24

Brian

6,391
3
33
49

Yes, I've ruled that out. I don't want ever to have to have the whole thing in memory. – jjujuma Jan 27 '11 at 11:27
Isnt EHCache payed ? Is there any free version – Vineeth Mohan Feb 08 '12 at 19:11
It's Open Source, so no license cost – Brian Feb 09 '12 at 09:13

score 0 · Answer 6 · answered Sep 18 '12 at 00:48

0

This seems like a relatively new open source solution to the problem, I've used it, and like it so far

https://github.com/jankotek/JDBM4

answered Sep 18 '12 at 00:48

Eran Medan

44,555
61
184
276

Are they any decent on-disk implementations of Java's Map?

6 Answers6

Linked

Related