1

I have seen lots of examples showing how to insert image data for model training in Caffe.

I am trying to train a model using data that is not images. I can reshape it as a matrix or a vector (for every example), but I don't understand how to make my Caffe network read it.

I know that Caffe can work with lmdb/hdf5 databases, and I can additionally use a Python data layer. I guess a Python data layer will be my best choice. Can someone provide an example of how to create some kind of array in Python and use it as training data for a Caffe model?

guyov
  • 93
  • 1
  • 7
  • if you are interested in an input python layer, you might find [this thread](http://stackoverflow.com/q/34996075/1714410) relevant. – Shai May 29 '16 at 06:49

1 Answers1

1

You do not need to create a python layer for simple vector inputs. HDF5 layer is probably easiest to work with. Just create HDF5 files with your favorite tool ( Refer this for creating HDF5 using matlab, or this for creating using python)

Both examples are fairly easy to follow. The matlab example gives you a more advanced version of HDF5 file creation--as in creating batches and all--but at its heart you just need to call 

store2hdf5(filename, data, labels) %others are optional

Similarly, the python example also goes over complete example, all of which you may or may not need. At its core, creating HDF5 file is simply.

import h5py
with h5py.File('filename.h5', 'w') as f:
    f['data'] = your_data
    f['label'] = your_labels

You can easily use the file thus created in HDF5 Datalayer easily as follows. You just need to create a text file with list of HDF5 files you want to use.

layer {
name: "data"
type: "HDF5Data"
top: "data"
top: "label"
include {
phase: TRAIN
}
hdf5_data_param {
source: "path_to_text_file_containing_list_of_HDF5_Files.txt" #
batch_size: 128
shuffle: true
}
}
Prophecies
  • 723
  • 1
  • 7
  • 19
  • Perfect! Could not ask for any better answer... So what is the use case of a python layer? – guyov May 24 '16 at 17:22
  • It's mainly used for writing functionalities that are beyond what is provided by caffe. If I answered your question, it would be great if you marked it as answered. – Prophecies May 26 '16 at 00:48
  • @Prophecies I do it similar to your example however I have an error: https://stackoverflow.com/questions/47930478/caffe-doesnt-read-the-hdf5-files-for-test-phase-instead-reads-the-path-given-i All the examples I have come across use "data" and "label" to name their tops but I have 3 tops so I have created specific names, it shouldn't be -but could it be the reason? If so how can I name them for 3 tops? – dusa Dec 22 '17 at 17:04