0

I am using @shai's code to convert data to hdf5, but after converting the size of data is too large, larger than limit size of caffe (2GB) so , my question is how we should split the data? we only need convert data separately depends on what we want ?

Soodabe Zarezade
  • 75
  • 1
  • 4
  • Did you make each example an hdf5 file and a single example is more than 2GB? Can you add a link to shai's code for reference? – Jonathan Jan 02 '17 at 15:42
  • @Jonathan I used matlab example code in caffe to convert my data, and i myself separate the data in a text file, but this shai's code : http://stackoverflow.com/q/31774953/6281477 – Soodabe Zarezade Jan 03 '17 at 08:48

1 Answers1

0

It would be helpful to have more information to give a better answer. Please give the actual error that Caffe gives you. Also, what exactly is larger than 2GB? Is it the txt file?

HDF5 is not very efficient. I would recommend either using the database layer (http://caffe.help/manual/layers/data.html) or writing your own customized Python layer (http://caffe.help/manual/layers/python.html).

Jonathan
  • 4,847
  • 3
  • 32
  • 37