1

I am not sure how to train large amount of data in Caffe.

I try to train network using 300k images for training data and 100k images for validation data. I am using HDF5 because I want to deal with float data. However, there is limitation of the hdf5 file size. I cannot deal with 300k images at once, so I divided with them.

I made train.txt like below.

D:/Dataset/train_0.hdf5
D:/Dataset/train_1.hdf5
D:/Dataset/train_2.hdf5
…
D:/Dataset/train_100.hdf5

and also make prototxt to call hdf5 files

layer {
  name: "data"
  type: "HDF5Data"
  top: "data"
  top: "label"
  hdf5_data_param {
    source: "C:/Users/…/data/train.txt"
    batch_size: 10
    shuffle: true
  }
  include {
    phase: TRAIN
  }
}

I am not sure this is right way. And I am also not sure how to set bath_size and test_iter, test_interval in solver. (ex. If the batch_size of Test is 10, is it correct to set 10k in test_iter? I think batch_size × test_iter has to be the number of validation images, but I think 10k is too large.)

I read the similar question(Caffe | solver.prototxt values setting strategy), and tried test_iter = 10k. But it didn't work well at that time(memory is not enough), so I thought I set the wrong value at test_iter(I set too large value). I am not sure how to decide batch_size and test_iter to the large dataset.

Masahiro
  • 155
  • 9

0 Answers0