How big can Erlang DETS be and what to do if its too small?

Question

All I need is a large persistent lookup table in Erlang and dets seems like just the thing though I need a definative answer to:

just how big the total size of the binaries in the table can be.
how big each entry can be
what to do if the answer to the first question is less than 100G

score 8 · Answer 1 · answered Nov 20 '09 at 05:59

8

This is kind of an RTFM question. As quoted directly in the second paragraph of the DETS manual:

The size of Dets files cannot exceed 2 GB. If larger tables are needed, Mnesia's table fragmentation can be used.

answered Nov 20 '09 at 05:59

Dustin

89,080
21
111
133

1

unless its 64 bit erlang or one of the other exceptions floating around. Which is why I ask here. – Arthur Ulfeldt Nov 20 '09 at 18:27
1

dets still have the limit on 64bit beams, its in the file format. – Christian Nov 20 '09 at 22:28

score 2 · Accepted Answer · answered Nov 20 '09 at 16:54

2

One obvious approach, once it is thought of, is to hash dets entries over multiple dets files.

A linear hash should make it dynamically growable, by splitting buckets into newly created dets files when one file reaches an upper threshold.

There are also a number of port drivers that enable you to use sleepycat/berkely db, or tokyo tyrrant. Those databases have file limits that are much higher than 2Gb.

answered Nov 20 '09 at 16:54

Christian

9,417
1
39
48

a good one here:http://dukesoferl.blogspot.com/2008/06/tokyocabinet-and-mnesia.html – Arthur Ulfeldt Nov 20 '09 at 21:58

score 0 · Answer 3 · answered Nov 20 '09 at 21:56

0

using mnesia fragmented dic_copies can overcome these limits provided you know how many fragments to crate ahead of time
http://www.trapexit.org/Mnesia_Table_Fragmentation

answered Nov 20 '09 at 21:56

Arthur Ulfeldt

90,827
27
201
284

How big can Erlang DETS be and what to do if its too small?

3 Answers3