Caffe's way of doing data shuffling

Question

Is shuffling done by setting the flag --shuffle as below as found in create_imagenet.sh ? :

GLOG_logtostderr=1 $TOOLS/convert_imageset \
   --resize_height=$RESIZE_HEIGHT \
   --resize_width=$RESIZE_WIDTH \
   --shuffle \

I mean I don't need to shuffle it manually afterwards, if the flag does it already. What about the label, is it shuffled automatically in the generated lmdb file?

if the data is shuffled and the labels are not, how can you use the resulting dataset for training? it becomes meaningless... — Shai, Feb 27 '17 at 13:58

score 1 · Accepted Answer · edited May 23 '17 at 12:16

Using convert_imageset tool creates a copy of your training/validation data in a binary database file (either in lmdb or leveldb format). The data encoded in the dataset includes pairs of example and its corresponding label.
Therefore, when shuffle-ing the dataset the labels are shuffled with the data to maintain the correspondence between data and its ground-truth label.
There is no need to shuffle the data again during training.

Caffe's way of doing data shuffling

1 Answers1