0

I need to create LMDBs dynamically that can be read by Caffe's data layer, and the constraint is that only C is available for doing so. No Python.

Another person examined the byte-level contents of a Caffe-ready LMDB file here: Caffe: Understanding expected lmdb datastructure for blobs

This is a good illustrative example but obviously not comprehensive. Drilling down led me to the Datum message type, defined by caffe.proto, and the ensuing caffe.pb.h file created by protoc from caffe.proto, but this is where I hit a dead end.

The Datum class in the .h file defines a method that appears to be a promising lead: 

void SerializeWithCachedSizes(::google::protobuf::io::CodedOutputStream* output) const

I'm guessing this is where the byte-level magic happens for encoding messages before they're sent.

Question: can anyone point me to documentation (or anything) that describes how the encoding works, so I can replicate an abridged version of it? In the illustrative example, the LMDB file contains MNIST data and metadata, and 0x08 seems to signify that the next value is "Number of Channels". And 0x10 and 0x18 designate heights and widths, respectively. 0x28 appears to designate an integer label being next. And so on, and so forth.

I'd like to gain a comprehensive understanding of all possible bytes and their meanings.

Shai
  • 111,146
  • 38
  • 238
  • 371
Pete Janda
  • 73
  • 8

1 Answers1

0

Additional digging yielded answers on the following page: https://developers.google.com/protocol-buffers/docs/encoding

Caffe.proto defines Datum by:

  • optional int32 channels = 1
  • optional int32 height = 2
  • optional int32 width = 3
  • optional bytes data = 4
  • optional int32 label = 5
  • repeated float float_data = 6
  • optional bool encoded = 7

The LMDB record's header in the illustrative example cited above is "08 01 10 1C 18 1C 22 90 06", so with the Google documentation's decoder ring, these hexadecimal values begin to make sense:

  • 08 = Field 1, Type = int32 (since tags are encoded by: (field_number << 3) | wire_type)
  • 01 = Value of Field 1 (i.e., number of channels) is 01
  • 10 = Field 2, Type = int32
  • 1C = Value of Field 2 (i.e., height) is 28
  • 18 = Field 3, Type = int32
  • 1C = Value of Field 3 (i.e., width) is 28
  • 22 = Field 4, Type = length-delimited in bytes
  • 90 06 = Value of Field 4 (i.e., number of bytes) is 1580 using the VarInt encoding methodology

Given this, efficiently creating LMDB entries directly with C for custom, non-image data sets that are readable by Caffe's data layer becomes straightforward.

Pete Janda
  • 73
  • 8