2

I am using a CNN network to classify images into 5 classes. The size of my dataset is around 370K. I am using Adam optimizer with learning rate 0.0001 and batch size of 32. Surprisingly, I am getting improvement in validation accuracy over the epochs but validation loss is constantly growing.

I am assuming that the model is becoming less and less unsure about the validation set but the accuracy is more because the value of softmax output is more than the threshold value.

What can be the reason behind this? Any help in this regard would be highly appreciated. Loss curve

Accuracy curve

Upendar Gareri
  • 427
  • 1
  • 4
  • 14
  • 1
    this is a case study of overfitting. as the training loss goes down, the validation loss is increasing – seralouk Jul 13 '18 at 09:28
  • yeah but in case of overfitting the validation accuracy should also be going down, right? – Upendar Gareri Jul 13 '18 at 09:36
  • 2
    It's not that simple - see [Loss & accuracy - Are these reasonable learning curves?](https://stackoverflow.com/questions/47817424/loss-accuracy-are-these-reasonable-learning-curves/47819022#47819022) for the subtle relation between loss & accuracy (although the underlying question there is different, I think you may find the answer helpful) – desertnaut Jul 13 '18 at 11:01

1 Answers1

1

I think this is a case of overfitting, as previous comments pointed out. Overfitting can be the result of high variance in the dataset. When you trained the CNN it showed a good ratio towards the decreasing of training error, producing a more complex model. More complex models produce overfitting and it can be noted when validation error tends to increase.

Adam optimizer is taking care of the learning rate, exponential decay and in general of the optimization of the model, but it won't take any action against overfitting. If you want to reduce it (overfitting), you will need to add a regularization technique which will penalize large values of the weights in the model.

You can read more details about this in the deep learning book: http://www.deeplearningbook.org/contents/regularization.html

Bitzel
  • 61
  • 4