Integer values in leveldb

Question

I would like to use leveldb to store integers and integer arrays efficiently, in both c++ and python.

For integers in C++, i could convert the int to a char array. Any suggestions for storing the int array in c++ and int and int array in python?

Answer 1

You could consider using a library like Google's protobuf (http://code.google.com/p/protobuf/) which is capable of (de)serializing structured data. For the case you mention, a repeated field would do the trick:

message List {
  repeated int64 val = 1;
}

Given the varint encoding used by protocol buffers (and depending on your range of values) this could be an efficient way to store the integers.

http://code.google.com/apis/protocolbuffers/docs/encoding.html#varints

It is tough to say more without knowing a bit more about your use case. How many integers are going to be stored per array on average? What is the range of integer values? etc.

Answer 2

for python, struct can be efficient. Here is an example using the ctypes leveldb interface from leveldb-py storing the value 1,2,3,4,5 (as an array of integers) in the database with key 100:

import leveldb,array,struct

#this assumes 32-bit unsigned integers in machine order
value=struct.pack('p',array('I',[1,2,3,4,5]))
key=struct.pack('I',100)

db=leveldb.DB("/path/to/db", create_if_missing=True)
db[key]=value

For more efficiency, import the specific functions (e.g. "from struct import pack") and use the lelveldb's WriteBatch class, if you have a bunch of writes to do.

Can integer keys / values be stored in LevelDB? suggests that a custom comparator may be needed, however this particular Python leveldb interface doesn't support that. https://plyvel.readthedocs.org/en/latest/ may be a better option.

Answer 3

For an int array in c++ you have to pass the int array as a char* to the Slice class, which you can then easily put to the leveldb database, e.g.

int myArray[3] = {1,2,3};
Slice valueSlice = Slice( (const char*) myArray, sizeof(myArray) );

This is basically valid for all types as well as custom classes, e.g.

MyClass* newObj = new MyClass();
Slice valueSlice = Slice( (const char*) newObj, sizeof(MyClass) );