I'm training a convolutional neural network using pylearn2 library and during all the ephocs, my validation error is consistently higher than the testing error. Is it possible? If so, in what kind of situations?
Asked
Active
Viewed 2,609 times
4
-
It's unusual. Are you using the validation set to optimize the hyperparameters, and the test set to test the true error at the end (which you should only do once, ideally.) What are the sizes of the training, validation, and test sets? – Houshalter May 22 '15 at 02:05
-
training set is 5000, validation set is 1000, training set is 2000 – SameeraR May 22 '15 at 06:26
2 Answers
5
moving the comment to an answer; modifying my previous answer seemed wrong
The full dataset may not be properly shuffled so the examples in the test set may be easier to classify.
Doing the experiment again with examples redistributed among the train / valid / test subsets would show if this is the case.
Nicu Tofan
- 1,052
- 14
- 34
0
Training set is a set of images that are fed to the network, errors are computed on the other end, then the parameters of the network are adjusted based on those errors. Validation set is a set of images that are fed to the network, errors are computed but parameters of the network are NOT adjusted.
Basically, you use validation to see how well the network performs on images it was not trained against.
In this view you should expect in most cases to have a higher error on valid_y_misclass than on train_y_miscalss.
See here for a discussion of the image sets.
Edit: example using pylearn2 notation
Size of train set: 700 examples; size of valid set: 300 examples
After some training (say 5 epochs) the network nails 650 out of 700 examples in the training set and 200 out of 300 in valid set.
As a result, after 5 epochs:
train_y_misclass = (700 - 650) / 700 = 0.07142857142
valid_y_misclass = (300 - 200) / 300 = 0.33333333333
valid_y_misclass > train_y_misclass and this is to be expected.
Community
- 1
- 1
Nicu Tofan
- 1,052
- 14
- 34
-
I agree. Thats the usual case. But is there anyway this could happen? I use the standard functions in pylearn 2 – SameeraR May 22 '15 at 06:27
-
Maybe I misread your question. You say that validation error is higher. I interpret that to mean that `valid_y_misclass` > `train_y_miscalss` which is the normal case. You mean that `valid_y_misclass` < `train_y_miscalss`? – Nicu Tofan May 22 '15 at 10:16
-
@TNick, he's saying that valid_y_misclass > test_y_misclass (using your notation, I'm not familiar with pylearn2.) Test sets are sometimes used to test if the hyperparameters are overfitting to the validation set. Which usually happens to some degree, which is why it is unusual. – Houshalter May 22 '15 at 15:33
-
@Houshalter I edited the answer; I'm certain that I'm missing something obvious; `valid_y_misclass > train_y_miscalss` indicates precisely that: the model is over-fitting the dataset to some degree (it is better at predicting the classes for the images it already saw and it is less good at predicting the classes for images it never saw before). – Nicu Tofan May 22 '15 at 16:16
-
@TNick. its not train_y_miscalss. Its test_y_miscalss. valid_y_misclass > test_y_miscalss. It's not normal right?? – SameeraR May 22 '15 at 18:28
-
And there's the obvious thing that I was missing. :) No, it's not; what are the actual values? And what if you shuffle the dataset (redistribute examples among train / valid / test)? Maybe there's something about the images in the test set that makes them particularly easy to classify. – Nicu Tofan May 22 '15 at 19:00
-
@TNick. Spot on. I havn't done data shuffling properly. Only negative samples existed in the testing set. Re-shuffling properly solved the issue. Can you post this as the answer so I can mark it. I know this is not a definite answer but in my case it was correct. – SameeraR May 24 '15 at 09:11
-
@TNick by the way you are not Nicu Tofan right? Sorry for this irrelevant comment. :) – SameeraR May 24 '15 at 09:36
-