Stack Overflow
Stack Overflow

Questions tagged [lmdb]

It is the mmap’d database for OpenLDAP aka Lightning MDB (LMDB).

It is the mmap’d database for OpenLDAP aka Lightning MDB (LMDB).

254 questions
17
votes
2 answers

Caffe: Reading LMDB from Python

I've extracted features using caffe, which generates a .mdb file. Then I'm trying to read it using Python and display it as a readable number. import lmdb lmdb_env = lmdb.open('caffefeat') lmdb_txn = lmdb_env.begin() lmdb_cursor =…
ytrewq
  • 3,670
  • 9
  • 42
  • 71
17
votes
3 answers

How to read image from numpy array into PIL Image?

I am trying to read an image from a numpy array using PIL, by doing the following: from PIL import Image import numpy as np #img is a np array with shape (3,256,256) Image.fromarray(img) and am getting the following error: File "...Image.py", line…
Aly
  • 15,865
  • 47
  • 119
  • 191
14
votes
3 answers

How do I count and enumerate the keys in an lmdb with python?

import lmdb env = lmdb.open(path_to_lmdb) Now I seem to need to create a transaction and a cursor, but how do I get a list of keys that I can iterate over?
Doug
  • 1,446
  • 4
  • 14
  • 26
12
votes
2 answers

Caffe: Understanding expected lmdb datastructure for blobs

I'm trying to understand how data is interpreted in Caffe. For that I've taken a look at the Minst Tutorial Looking at the input data definition: layers { name: "mnist" type: DATA data_param { source: "mnist_train_lmdb" backend: LMDB …
Sam7
  • 3,382
  • 2
  • 34
  • 57
10
votes
1 answer

What is special about internal design of LMDB?

What would be the performance difference (reads/writes) between some C++ implementation of in-memory B-Tree (for example google btree) and the LMDB (without taking into consideration all the feacures of LMDB such as transactions, isolation, shared…
pavelkolodin
  • 2,859
  • 3
  • 31
  • 74
9
votes
1 answer

How to force 64-bit LMDB to generate a 32-bit database?

The executable which uses the lmdb library is in 64 bit linux format . Because dependency libraries of the executable are in 64 bit .Building executable & lmdb lib as 32bit solution not possible. Is there any way i can force 64 bit lmdb lib to…
Rajesh Gopu
  • 863
  • 9
  • 33
9
votes
3 answers

Generating LMDB for Caffe

I am trying to build a deep learning model for Saliency analysis using caffe (I am using the python wrapper). But I am unable to understand how to generate the lmdb data structure for this purpose. I have gone through the Imagenet and mnist examples…
Unni
  • 5,348
  • 6
  • 36
  • 55
9
votes
4 answers

Writing data to LMDB with Python very slow

Creating datasets for training with Caffe I both tried using HDF5 and LMDB. However, creating a LMDB is very slow even slower than HDF5. I am trying to write ~20,000 images. Am I doing something terribly wrong? Is there something I am not aware…
Simikolon
  • 317
  • 3
  • 9
8
votes
4 answers

Count number of records in lmdb databse with python

I open a lmdb database using this code: lmdb_env = lmdb.open(source_path, readonly=True) How can I count the number of records in this database?
trminh89
  • 877
  • 2
  • 10
  • 17
8
votes
1 answer

Multi-label classification with Caffe

I want to fine-tune GoogLeNet to do Multi-label classification with Caffe. I have already fine-tuned it to a single-label classification, but I can't make the transition to Multi-label yet. The main steps I am doing which are different: Create LMDB…
YotamH
  • 95
  • 8
8
votes
0 answers

In LMDB, can multiple threads simultaneously access the same database?

I'm considering using LMDB in a web application, which, of course, will be multi-threaded. In they Symas documentation for mdb_dbi_open it states "This function must not be called from multiple concurrent transactions. A transaction that uses this…
Steve Schafer
  • 91
  • 1
  • 3
7
votes
1 answer

LMDB random writes really slow for large data (~1MB/s)

I am testing LMDB performance with the benchmark given in http://www.lmdb.tech/bench/ondisk/. And I noticed that LMDB random writes are really slow when the data goes beyond memory. I am using a machine with 4GB DRAM with Intel PCIE SSD. The key…
hjk41
  • 239
  • 2
  • 10
7
votes
1 answer

lmdb: how to determine free space left?

When creating lmdb environment I can specify map size. Is there a way to determine at any point how much of the map size is used up? In other words, I need to find out how much free space is left to be able to address the issue of running out of…
Sumrak
  • 3,740
  • 7
  • 31
  • 37
7
votes
1 answer

how to append data to existing LMDB?

I have around 1 million images to put in this dataset 10000 at a time appended to the set. I"m sure the map_size is wrong with ref from this article used this line to create the set env = lmdb.open(Path+'mylmdb', map_size=int(1e12) use this line…
Arsenal Fanatic
  • 3,663
  • 6
  • 38
  • 53
6
votes
1 answer

Unable to make field long java.nio.Buffer.address accessible: module java.base does not "opens java.nio" - LMDB using Java API

I'm trying to create DB and env using LMDB. I'm facing an issue on Env. create(). I have used LMDB documentation for this. Exception in thread "main" java.lang.ExceptionInInitializerError at…
Vishnu Chithan
  • 76
  • 2
  • 6
1
2 3
…
16 17