How to Create HDF5 training data for stereo networks out of image directory?

Question

This question refers to a question answered here.
The accepted answer suggests to create labels on the fly. I have a very similar problem but need to use HDF5.

Here is my prototxt:

name: "StereoNet"
layer {
    name: "layer_data_left"
    type: "HDF5Data"
    top: "data_left"
    top: "labels_left"
    include {
        phase: TRAIN
    }
    hdf5_data_param {
        source: "/home/ubuntu/trainLeftPatches.txt"
        batch_size: 128
    }
}
layer {
    name: "layer_data_right"
    type: "HDF5Data"
    top: "data_right"
    top: "labels_right"
    include {
        phase: TRAIN
    }
    hdf5_data_param {
        source: "/home/ubuntu/trainRightPatches.txt"
        batch_size: 128
    }
}
...  etc.

As you hopefully understand, I create two separate data HDF5 data files. They consist of positive and negative samples by having on the same index a left and a right image that in combination are a positive or negative sample. The labels_left and labels_right are identical matlab arrays of 1's and 0's. I tried to use a single labels array before but caffe gave an error, which seemed to indicate that two processes were clashing. When changing to a copy of the labels array, the training could start.

Here is part of the Matlab data creation file I am now using, the data are the KITTI data:

h5create('trainLeftPatches.h5','/data_left',[9 9 1 numberOfTrainingPatches],'Datatype','double');
h5create('trainLeftPatches.h5','/labels_left',[1 numberOfTrainingPatches],'Datatype','double');

h5create('trainRightPatches.h5','/data_right',[9 9 1 numberOfTrainingPatches],'Datatype','double');
h5create('trainRightPatches.h5','/labels_right',[1 numberOfTrainingPatches],'Datatype','double');

h5create('valLeftPatches.h5','/data_left',[9 9 1 numberOfValidatePatches],'Datatype','double');
h5create('valLeftPatches.h5','/labels_left',[1 numberOfValidatePatches],'Datatype','double');

h5create('valRightPatches.h5','/data_right',[9 9 1 numberOfValidatePatches],'Datatype','double');
h5create('valRightPatches.h5','/labels_right',[1 numberOfValidatePatches],'Datatype','double');

h5write('trainLeftPatches.h5','/data_left', dataLeft_permutated(:, :, :, 1:numberOfTrainingPatches));
h5write('trainLeftPatches.h5','/labels_left', labels_permutated(:, 1:numberOfTrainingPatches));

h5write('trainRightPatches.h5','/data_right', dataRight_permutated(:, :, :, 1:numberOfTrainingPatches));
h5write('trainRightPatches.h5','/labels_right', labels_permutated(:, 1:numberOfTrainingPatches));

h5write('valLeftPatches.h5','/data_left', dataLeft_permutated(:, :, :, numberOfTrainingPatches+1:end));
h5write('valLeftPatches.h5','/labels_left', labels_permutated(:, numberOfTrainingPatches+1:end));

h5write('valRightPatches.h5','/data_right', dataRight_permutated(:, :, :, numberOfTrainingPatches+1:end));
h5write('valRightPatches.h5','/labels_right', labels_permutated(:, numberOfTrainingPatches+1:end));

toc;

the loss is acceptable on mini batches at the end, but stays too high on the tests

Please advice. (It may not work). If there is an error, it is probably very subtle.

in matlab, for creating file names it is better to use `fullfile` command than using `strcat`. — Shai, Jun 14 '16 at 06:15

score 0 · Answer 1 · answered Jun 14 '16 at 06:23

0

A few points you should consider:

Your network is not a Siamese network: it contains two paths left and right, but these paths do not share the same filters. See this tutorial how to build a siamese network that shares filters across layers.
"HDF5Data" layer is not restricted to two outputs ("top"s), it can have as many as you'd like. Thus, you can have a single layer for train and a single layer for test:
```
layer {
  name: "data"
  type: "HDF5Data"
  top: "data_left"
  top: "data_right"
  top: "labels"
  hdf5_data_param { ... }
  include { phase: TRAIN }
}
```
The corresponding hdf5 files should have three dataset specified for the h5write command (instead of only two in your code).
Have you considered using minibatch loss, instead of pairs loss?

answered Jun 14 '16 at 06:23

Shai

111,146
38
238
371

1

Hi Shai, thank you, I will correct (1) indeed it is supposed to be a 'stereo network', redoing https://arxiv.org/abs/1409.4326. For (2) I will try using a single HDF5 file and then see if it works or how it fares with (3). – imonaboat Jun 14 '16 at 06:38
F0614 07:26:56.565969 27540 net.cpp:405] Top blob 'labels' produced by multiple sources. *** Check failure stack trace: *** That's when I put training and validate patches in one file with a single labels H5 array. – imonaboat Jun 14 '16 at 07:27
@imonaboat you cannot have two layers producing `"label"` at the same `phase`. – Shai Jun 14 '16 at 07:29
No, but I changed the H5 files and changed the prototxt to your suggestion but not the network layout. I guess that what you say is "can't have two layers produce label at the same phase", is right and, it is the reason I copied the label array to two separate h5 files. (In the code here there were also two errors from last night. I just corrected them. So it is a stereo network, not a siamese one.) – imonaboat Jun 14 '16 at 08:47
Do you understand? - there are multiple ways to avoid 'producing label at two phases'. I need to keep my network layout and make sure that the data on the left and right are fed to the network 'simultaneously'. That is, read a left blob(i) + label(i) and read a right blob(i) + label(i) at the same time, keeping them together, as in stereo vision. – imonaboat Jun 14 '16 at 08:54
@imonaboat I am not familiar with "stereo network", but you can use `"HDF5Data"` with `top: "data_right" top: "data_left" top: "label_right" top: "label_left"` for a single layer. Thus, you sync the left/right data/labels during creation of hdf5 files and you do not need to worry about it during training/testing. – Shai Jun 14 '16 at 09:03
@imonaboat another minor comment, I couldn't follow your matlab code for creating the hdf5 files, but are you shuffling the examples? see [this thread](http://stackoverflow.com/q/37658069/1714410). – Shai Jun 14 '16 at 09:03
yes, they are shuffled by the permutationVector, I cannot use a single data layer, because the network is specified by the article, you could step through the code using Matlab and the KITTI dataset, here is a link: http://kitti.is.tue.mpg.de/kitti/data_scene_flow.zip (if you are interested or please) – imonaboat Jun 14 '16 at 09:13
Perhaps I should note that even with 840000 samples and 600000 iterations, the runs do give fairly similar but still varied results. Many of them, especially after different datasets have been created using the script here, do not reach good iteration marks, nor test marks. – imonaboat Jun 14 '16 at 09:33
maube the last datasample() is not proper and I should select the head of that array by the same length of count_dpr – imonaboat Jun 14 '16 at 11:30

How to Create HDF5 training data for stereo networks out of image directory?

1 Answers1