4

I have the train and label data as data.mat. (I have 200 training data with 6000 features and labels are (-1, +1) that have saved in data.mat).

I am trying to convert my data in hdf5 and run Caffe using:

load data.mat
hdf5write('my_data.h5', '/new_train_x', single( reshape(new_train_x,[200, 6000, 1, 1]) ) );
hdf5write('my_data.h5', '/label_train', single( reshape(label_train,[200, 1, 1, 1]) ), 'WriteMode', 'append' );

And my layer.prototxt (just data layer) is:

layer {
  type: "HDF5Data"
  name: "data"
  top: "new_train_x"     # note: same name as in HDF5
  top: "label_train"     # 
  hdf5_data_param {
    source: "/path/to/list/file.txt"
    batch_size: 20
  }
  include { phase: TRAIN }
}

but, i have an error: ( Check failed: hdf_blobs_[i]->shape(0) == num (200 vs. 6000))

I1222 17:02:48.915861  3941 layer_factory.hpp:76] Creating layer data
I1222 17:02:48.915871  3941 net.cpp:110] Creating Layer data
I1222 17:02:48.915877  3941 net.cpp:433] data -> new_train_x
I1222 17:02:48.915890  3941 net.cpp:433] data -> label_train
I1222 17:02:48.915900  3941 hdf5_data_layer.cpp:81] Loading list of HDF5 filenames from: file.txt
I1222 17:02:48.915923  3941 hdf5_data_layer.cpp:95] Number of HDF5 files: 1
F1222 17:02:48.993865  3941 hdf5_data_layer.cpp:55] Check failed: hdf_blobs_[i]->shape(0) == num (200 vs. 6000) 
*** Check failure stack trace: ***
    @     0x7fd2e6608ddd  google::LogMessage::Fail()
    @     0x7fd2e660ac90  google::LogMessage::SendToLog()
    @     0x7fd2e66089a2  google::LogMessage::Flush()
    @     0x7fd2e660b6ae  google::LogMessageFatal::~LogMessageFatal()
    @     0x7fd2e69f9eda  caffe::HDF5DataLayer<>::LoadHDF5FileData()
    @     0x7fd2e69f901f  caffe::HDF5DataLayer<>::LayerSetUp()
    @     0x7fd2e6a48030  caffe::Net<>::Init()
    @     0x7fd2e6a49278  caffe::Net<>::Net()
    @     0x7fd2e6a9157a  caffe::Solver<>::InitTrainNet()
    @     0x7fd2e6a928b1  caffe::Solver<>::Init()
    @     0x7fd2e6a92c19  caffe::Solver<>::Solver()
    @           0x41222d  caffe::GetSolver<>()
    @           0x408ed9  train()
    @           0x406741  main
    @     0x7fd2e533ca40  (unknown)
    @           0x406f69  _start
Aborted (core dumped)

Many thanks!!!! Any advice would be appreciated!

Shai
  • 111,146
  • 38
  • 238
  • 371
ahmad navid ghanizadeh
  • 139
  • 9
  • can you [`h5ls`](https://www.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Ls) the file `my_data.h5`? – Shai Dec 22 '15 at 14:54
  • I suspect something along the lines of [this](http://www.mathworks.com/help/matlab/ref/h5d.write.html): *"Note: The HDF5 library uses C-style ordering for multidimensional arrays, while MATLAB uses FORTRAN-style ordering. Please consult "Using the MATLAB Low-Level HDF5 Functions" in the MATLAB documentation for more information."* – Shai Dec 22 '15 at 15:00
  • @Shai , I suppose so, i check it and be aware you. many thanks. – ahmad navid ghanizadeh Dec 22 '15 at 16:37

1 Answers1

3

The problem

It seems like there is indeed a conflict with the order of elements in arrays: matlab arranges the elements from the first dimension to the last (like fortran), while caffe and hdf5 stores the arrays from last dimension to first:
Suppose we have X of shape nxcxhxw then the "second element of X" is X[2,1,1,1] in matlab but X[0,0,0,1] in C (1-based vs 0-based indexing doesn't make life easier at all).
Therefore, when you save an array of size=[200, 6000, 1, 1] in Matlab, what hdf5 and caffe are actually seeing is as array of shape=[6000,200].

Using the h5ls command line tool can help you spot the problem.
In matlab you saved

>> hdf5write('my_data.h5', '/new_train_x', 
  single( reshape(new_train_x,[200, 6000, 1, 1]) );
>> hdf5write('my_data.h5', '/label_train', 
  single( reshape(label_train,[200, 1, 1, 1]) ),
  'WriteMode', 'append' );

Now you can inspect the resulting my_data.h5 using h5ls (in Linux terminal):

user@host:~/$ h5ls ./my_data.h5
  label_train              Dataset {200}
  new_train_x              Dataset {6000, 200}

As you can see, the arrays are written "backwards".

Solution

Taking this conflict into account when exporting data from Matlab, you should permute:

load data.mat
hdf5write('my_data.h5', '/new_train_x', 
  single( permute(reshape(new_train_x,[200, 6000, 1, 1]),[4:-1:1] ) );
hdf5write('my_data.h5', '/label_train', 
  single( permute(reshape(label_train,[200, 1, 1, 1]), [4:-1:1] ) ),
  'WriteMode', 'append' );

Inspect the resulting my_data.h5 using h5ls now results with:

user@host:~/$ h5ls ./my_data.h5
  label_train              Dataset {200, 1, 1, 1}
  new_train_x              Dataset {200, 6000, 1, 1}

Which is what you expected in the first place.

Community
  • 1
  • 1
Shai
  • 111,146
  • 38
  • 238
  • 371
  • ,I do it :) . But i have another problem. in inner product layer, in bottom blob i enter ''new_train_x'', but it give an error. F1224 20:30:43.874462 22511 insert_splits.cpp:35] Unknown bottom blob 'new_train_x' (layer 'ip1', bottom index 0) *** Check failure stack trace: ***. what should i do?! thanks much. – ahmad navid ghanizadeh Dec 24 '15 at 14:01
  • layer { name: "ip1" type: "InnerProduct" bottom: "new_train_x" top: "ip1" param { lr_mult: 1 } param { lr_mult: 2 } inner_product_param { num_output: 30 weight_filler { type: "gaussian" # initialize the filters from a Gaussian std: 0.01 } bias_filler { type: "constant" } } } – ahmad navid ghanizadeh Dec 24 '15 at 14:02
  • @ahmadnavidghanizadeh sounds like you have a **new** question. please post it as such. It is impossible to read code in comments. – Shai Dec 24 '15 at 14:09
  • ok. i post it [this](http://stackoverflow.com/questions/34454278/check-fail-how-to-use-hdf5-data-layer-in-deep-layer). thanks a lot. – ahmad navid ghanizadeh Dec 24 '15 at 14:58
  • @Shai I don't understand one thing here, we want caffe to read data as n * c * h * w, first here i understand that c (the number of channels) is 1 and w is 6000 (the number of features), but from your answer i understand that c is 6000 while w is 1. Am I missing something here? – Deven Feb 25 '16 at 00:01
  • @Deven the channel dimension is the feature dimension. each sample is 1by1 feature spatially with dim 6000 – Shai Feb 25 '16 at 05:47
  • I still do not undertstand, can you be please more elaborate. For an image channel dimension = 3, so for a 2D matrix as input feature matrix, (say like a matrix representing a sentence with w words and k dimensional word embedding, input matrix will be w*k) shouldn't channel dimension be 1? – Deven Feb 25 '16 at 09:29
  • @Deven since there is no "width" and "height" for a sentence with `w` words embedded in `k` dimensions it is quite nonsense to force it into a 4D data structure. So, what I am saying is just a convention (and not too established one at that). So, if it works better to you to `reshape` it differently - then go ahead and do so. – Shai Feb 25 '16 at 09:39
  • Ok, I assume that then the filter shapes would also correspondingly change? – Deven Feb 25 '16 at 12:14