0

I'm wanting to use the rust implementation of LMDB but I can't seem to find whether it would always maintain a copy of whats in memory also in the disk and viceversa. My reasoning would be that the DB will have some cache and any overflows would be written to disk but since I can't find it in the documentation I'm not sure.

The other case would be that LMDB maps its contents on disk to memory and that would work for small quantities but not for what I have in mind.

Does anyone have an idea on how LMDB works on this regard?

amirouche
  • 7,682
  • 6
  • 40
  • 94
CastleHanks
  • 3
  • 1

2 Answers2

2

Does anyone have an idea on how LMDB works on this regard?

If you are worried about not being able to operate on a dataset that does not fit in memory, you are ok - LMDB does handle that. Your dataset is not limited to the size of ram.

LMDB is memory-mapped which is a technique that allows developers to read/write data on disk 'like' it is in memory - the OS does all the heavy lifting required

LMDB always stored BOTH in disk AND memory?

The entire dataset is on disk. Some parts of it are in memory. When the parts that are not in memory are needed - the OS fetches it from disk and gives it to the application by putting it in the process' memory

Asad Awadia
  • 1,417
  • 2
  • 9
  • 15
  • Yes your response is correct, but loading from disk is slow. That is not the intended use case of LMDB [citation needed]. – amirouche Apr 01 '22 at 14:33
0

The other case would be that LMDB maps its contents on disk to memory and that would work for small quantities but not for what I have in mind.

Yes, that is it.

I read RocksDB support the usage you are looking while offering similar guarantees.

Small quantities depends on the available RAM. Major vendors including mongodb with wiredtiger backend, but not only, postgresql comes also to mind, that highly recommend to have as much memory as the working dataset.

In an opposite, you can find database systems such as rust full text search engine using an cold (offline?) object storage.

amirouche
  • 7,682
  • 6
  • 40
  • 94