Is there a good port of leveldb for C#?

Question

I wish to use leveldb in my pure C# project.

I have googled for a C# version of leveldb, but got no lucky.

Any one can tell me where I can find a C# version of leveldb?

Thanks

Answer 1

Not that I know of, but I've been using it in my C# project. If you're familiar with C++, then you can make your own CLI wrapper (shouldn't be that much trouble), build it as a DLL and then you can load that DLL in your C# project like any other assembly reference.

There is a windows port for leveldb and it's a little tricky to get it into Visual Studio, but if you're having trouble I can upload my Visual Studio 2010 solution (which is 75% of the battle) with the whole thing set-up and ready to build (except the CLI wrapper). I can put it up on github or something, which I'm actually planning on doing anyway, but I'll expedite it for you.

Like I said, I've been using that approach for my C# project and it works great. However, if you have really high performance requirements, then I would recommend batching up "work" in order to reduce the P/Invokes.

Example

Please note that I have not compiled this code, but I'm just posting it as an example. Your header file might look like this:

#pragma once
#include <exception>
#include "leveldb\db.h"

using namespace System::Runtime::InteropServices;

// Create the namespace
namespace LevelDBWrapperNS 
{
    // Note that size_t changes depending on the target platform of your build:
    // for 32-bit builds, size_t is a 32-bit unsigned integer.
    // for 64-bit builds, size_t is a 64-bit unsigned integer.
    // There is no size_t equivalent in C#, but there are ways to
    // mimic the same behavior. Alternately, you can change the
    // size_t to unsigned long for 32-bit builds or unsigned long long (64-bit)

    // Declare the leveldb wrapper
    public ref class LevelDBWrapper
    {
    private:
        leveldb::DB*        _db;
    public:
        LevelDBWrapper(const std::string dataDirectory);
        ~LevelDBWrapper();

        // A get method which given a key, puts data in the value array
        // and sets the valueSize according to the size of the data it
        // allocated. Note: you will have to deallocate the data in C#
        void Get(const char* key, const size_t keySize, char* value, size_t &valueSize);

        // A put method which takes in strings instead of char*
        bool Put(const std::string key, const std::string value);

        // A put method which takes in char* pointers
        bool Put(const char* key, const size_t keySize, const char* value, const size_t valueSize);

        // A delete method
        bool Delete(const char* key, const size_t keySize);

    private:
        void Open(const char* dataDirectory);
    };
}

Your cpp file is going to be along the lines of:

#include "LevelDBWrapper.h"

// Use the same namespace as the header
namespace LevelDBWrapperNS
{
    LevelDBWrapper::LevelDBWrapper(const std::string dataDirectory)
    {
        Open(dataDirectory.c_str());
    }

    LevelDBWrapper::~LevelDBWrapper()
    {
        if(_db!=NULL)
        {
            delete _db;
            _db= NULL;
        }

        // NOTE: don't forget to delete the block cache too!!!
        /*if(options.block_cache != NULL)
        {
            delete options.block_cache;
            options.block_cache = NULL;
        }*/
    }

    bool LevelDBWrapper::Put(const char* key, const size_t keySize, const char* value, const size_t valueSize)
    {
        leveldb::Slice sKey(key, keySize);
        leveldb::Slice sValue(value, valueSize);

        return _db->Put(leveldb::WriteOptions(), sKey, sValue).ok();
    }

    void LevelDBWrapper::Open(const char* dataDirectory)
    {
        leveldb::Options    options;

        // Create a database environment. This will enable caching between 
        // separate calls (and improve performance). This also enables 
        // the db_stat.exe command which allows cache tuning. Open 
        // transactional environment leveldb::Options options;
        options.create_if_missing = true;

        // Open the database if it exists
        options.error_if_exists = false;

        // 64 Mb read cache
        options.block_cache = leveldb::NewLRUCache(64 * 1024 * 1024);   

        // Writes will be flushed every 32 Mb
        options.write_buffer_size = 32 * 1024 * 1024;   

        // If you do a lot of bulk operations it may be good to increase the 
        // block size to a 64k block size. A power of 2 block size also 
        // also improves the compression rate when using Snappy.
        options.block_size = 64 * 1024; 
        options.max_open_files = 500;
        options.compression = leveldb::kNoCompression;

        _db = NULL;

        // Open the database
        leveldb::Status status = leveldb::DB::Open(options, dataDirectory, &_db);

        // Check if there was a failure
        if(!status.ok())
        {
            // The database failed to open!
            if(status.ToString().find("partial record without end")!=std::string::npos)
            {
                // Attempting to recover the database...

                status = leveldb::RepairDB(dataDirectory, options);

                if(status.ok())
                {
                    // Successfully recovered the database! Attempting to reopen... 
                    status = leveldb::DB::Open( options, dataDirectory, &_db);
                }
                else
                {
                    // Failed to recover the database!
                }
            }

            // Throw an exception if the failure was unrecoverable!
            if(!status.ok())
            {
                throw std::runtime_error(std::string("Unable to open: ") + std::string(dataDirectory) + 
                    std::string(" ") + status.ToString());
            }
        }
    }
}

This should get you in the right direction.

Get Example

OK, Get will look like this:

// Returns a buffer containing the data and sets the bufferLen.
// The user must specify the key and the length of the key so a slice
// can be constructed and sent to leveldb.
const unsigned char* Get(const char* key, const size_t keyLength, [Out]size_t %bufferLen);

The source is along the lines:

const unsigned char* LevelDBWrapper::Get(const char* key, const size_t keyLength, [Out]size_t %bufferLen)
{
    unsigned char* buffer = NULL;
    std::string value;
    leveldb::Status s = db->Get(leveldb::ReadOptions(), Slice(key, keyLength), &value);
    if(s.ok())
    {
        // we found the key, so set the buffer length 
        bufferLen = value.size();

        // initialize the buffer
        buffer = new unsigned char[bufferLen];

        // set the buffer
        memset(buffer, 0, bufferLen);

        // copy the data
        memcpy(memcpy((void*)(buffer), value.c_str(), bufferLen);
    }
    else
    {
        // The buffer length is 0 because a key was not found
        bufferLen = 0;
    }
    return buffer;
}

Note that different data may have different encoding, so I feel like the safest way to pass data between your unmanaged and managed code is to use pointers and an UnmanagedMemoryStream. Here is how you would get the data associated with a key in C#:

UInt32 bufferLen = 0;
byte* buffer = dbInstance.Get(key, keyLength, out bufferLen);
UnmanagedMemoryStream ums = new UnmanagedMemoryStream(buffer, (Int32)bufferLen, (Int32)bufferLen, FileAccess.Read);

// Create a byte array to hold data from unmanaged memory.
byte[] data = new byte [bufferLen];

// Read from unmanaged memory to the byte array.
readStream.Read(data , 0, bufferLen);

// Don't forget to free the block of unmanaged memory!!!
Marshal.FreeHGlobal(buffer);

Again, I have not compiled or run the code, but it should get you on the right track.

Answer 2

As much as I can see you could also use LMDB (lightning memory mapped database, http://symas.com/mdb/ ) which seems quite similar to LevelDB and also comes with a .Net wrapper (https://github.com/ilyalukyanov/Lightning.NET) Dont know how well it works though, haven't used it yet...

Answer 3

I haven't used it, but I see leveldb-sharp.

Answer 4

I don't know the story here, but there's this project at Microsoft's official Rx-Js page here.