2

I used hdf5 file as caffe input data, and the error occured:

hdf5_data_layer.cpp:75] Check failed: !this->layer_param_.has_transform_param() HDF5Data does not transform data.

This is my defination:

layer {
  name: "weight28"
  type: "HDF5Data"
  include { phase : TRAIN } 
  transform_param { scale: 0.00392156862745098 }
  hdf5_data_param {
    source: "/home/zhangyu/codes/unsupervised/data/weight28.h5"
    batch_size: 8
  }
  top: "weight28"
}

Here is some information of my h5 file:

HDF5 weight28.h5 
Group '/' 
    Dataset 'data' 
        Size:  2555000x28
        MaxSize:  Infx28
        Datatype:   H5T_IEEE_F64LE (double)
        ChunkSize:  28x28
        Filters:  none
        FillValue:  0.000000

I find a similar question and the answer said You cannot use transform param in the hdf5data layer.
What dose data transform in caffe do? Can I just cancel it?

Community
  • 1
  • 1
Zhang Yu
  • 559
  • 6
  • 15

2 Answers2

1

In response to the error mentioned by user6931072 Check failed: H5LTfind_dataset(file_id, dataset_name_) Failed to find HDF5 dataset:

You should provide a text file as the source, not an hdf5 file. So, instead of source: "/home/zhangyu/codes/unsupervised/data/weight28.h5", make a text file that lists the hdf5 file and then use source: "/home/zhangyu/codes/unsupervised/data/myhdf5data.txt".

meicholtz
  • 76
  • 1
  • 6
0

  1. As you already spotted yourself, you cannot have transformation_param in a "HDF5Data" layer - caffe does not support this.

  2. As for the transformation parameters themselves, look at caffe.proto:

// For data pre-processing, we can do simple scaling and subtracting the
// data mean, if provided. Note that the mean subtraction is always carried
// out before scaling.
optional float scale = 1 [default = 1];

Having transform_param { scale: 0.00392156862745098 } means your net expects your input to be scaled by 0.0039.. (1/254).
You can (and probably should) scale the data by 1/254 when you create the hdf5 data files for training and then remove the transform_param from the "HDF5Data" layer.

Shai
  • 111,146
  • 38
  • 238
  • 371
  • Thanks a lot, I removed the transform parameter and solved that. But it came with this error again: `Check failed: H5LTfind_dataset(file_id, dataset_name_) Failed to find HDF5 dataset`. Is there something wrong with my preparing of the hdf5 file? I was intended to prepare a dataset of 91250x28x28 which was feeded to the net together with other data defined in leveldb. @Shai – Zhang Yu Oct 15 '16 at 11:44
  • @user6931072 sounds like a new question to me. from the error message it seems like caffe cannot find the files. – Shai Oct 15 '16 at 16:33