4

I am now implementing neural network for classification. I use backpropagation algorithm to train. I use Cross validation method. But I am not clear when should I stop training the neural network.
Next one is how to check overfitting and underfitting.

I have a data set which has 1,000 pattern. I use 10 fold cross validation method. So 1 Fold has 100 pattern. I train with 900 pattern and test with 100 pattern.

Although I change no of hidden nodes and no of epoch, testing accuracy is not change very much. But I feed training data to trained network, accuracy of training is vary according to no of hidden nodes and no of epoch. Is My idea enough for checking overfitting and underfitting? Can I determine overfitting and underfitting only with accuracy?

I want to also ask some question continue with this. I post my result, that test with various hidden nodes and various no of epoch. As I told, I use cross validation, I use only one network(which get maximum on test accuracy) from 10 trained networks.

No of hidden nodes=50 ,Learning Rate=0.1 , no of epoch=100
Network 0 on Test=75.0 , onTrain= 97.11111111111111
Network 1 on Test=72.0 , onTrain= 98.22222222222223
Network 2 on Test=69.0 , onTrain= 97.88888888888889
> Network 3 on Test=78.0 , onTrain= 97.44444444444444
Network 4 on Test=77.0 , onTrain= 97.77777777777777
Network 5 on Test=77.0 , onTrain= 97.11111111111111
Network 6 on Test=69.0 , onTrain= 97.55555555555556
Network 7 on Test=74.0 , onTrain= 98.22222222222223
Network 8 on Test=76.0 , onTrain= 97.77777777777777
Network 9 on Test=74.0 , onTrain= 97.55555555555556

No of hidden nodes=50 ,Learning Rate=0.1 , no of epoch=70
Network 0 on Test=71.0 , onTrain= 93.22222222222221
Network 1 on Test=70.0 , onTrain= 93.33333333333333
Network 2 on Test=76.0 , onTrain= 89.88888888888889
Network 3 on Test=80.0 , onTrain= 93.55555555555556
Network 4 on Test=77.0 , onTrain= 93.77777777777779
> Network 5 on Test=81.0 , onTrain= 92.33333333333333
Network 6 on Test=77.0 , onTrain= 93.0
Network 7 on Test=73.0 , onTrain= 92.33333333333333
Network 8 on Test=75.0 , onTrain= 94.77777777777779
Network 9 on Test=70.0 , onTrain= 93.11111111111111

No of hidden nodes=50 ,Learning Rate=0.1 , no of epoch=50

Network 0 on Test=73.0 , onTrain= 87.8888888888889
Network 1 on Test=74.0 , onTrain= 89.22222222222223
Network 2 on Test=73.0 , onTrain= 87.1111111111111
Network 3 on Test=66.0 , onTrain= 90.44444444444444
Network 4 on Test=82.0 , onTrain= 88.77777777777777
Network 5 on Test=80.0 , onTrain= 88.44444444444444
Network 6 on Test=67.0 , onTrain= 88.33333333333333
Network 7 on Test=75.0 , onTrain= 87.8888888888889
Network 8 on Test=78.0 , onTrain= 87.44444444444444
Network 9 on Test=73.0 , onTrain= 85.0

The First Network( no of epoch=100) best network get accuracy on Test is 78.0 but on train is 97.4444. Is it mean overfitting? If this is overfitting, is the Third Network( no of epoch=50) best network get accuracy on test is 82.0 and on train is 88.777 acceptable? If not acceptable, should I decrease no of epoch?

Mel
  • 5,837
  • 10
  • 37
  • 42
Nay Zaw
  • 85
  • 4
  • 9

1 Answers1

3

See this answer for more details: whats is the difference between train, validation and test set, in neural networks?

OR if you like pseudo code, this is approximately what it would look like:

for each epoch
    for each training data instance
        propagate error through the network
        adjust the weights
        calculate the accuracy over training data
    for each validation data instance
        calculate the accuracy over the validation data
    if the threshold validation accuracy is met
        exit training
    else
        continue training
Community
  • 1
  • 1
Kiril
  • 39,672
  • 31
  • 167
  • 226
  • I think this may ok with Holdout Validation Method. Is it ok for Cross Validation? I use 10 Fold Cross Validation. So I think, I can't stop the training before complete training all of the 10 networks. If u ok, pls help me for 10 Fold Cross Validation Method. If I use cross validation ,Do I need checking overfitting and underfitting ? thanks sir – Nay Zaw Sep 23 '11 at 03:21
  • You can replace the holdout validation with 10-fold cross validation, I don't see a problem with that idea. If your training accuracy keeps rising but your cross-validation accuracy remains the same, then you should stop training because you're probably over-fitting. – Kiril Sep 23 '11 at 03:29