keras CNN : train and validation set are identical but with different accuracy

Question

I know this a very bad thing to do but I noticed something strange using keras mobilenet :

I use the same data for training and validation set :

train_generator = train_datagen.flow_from_directory(
    train_dir,
    target_size=(IM_WIDTH, IM_HEIGHT),
    batch_size=batch_size,
    class_mode = "categorical"
)
validation_generator = train_datagen.flow_from_directory(
  train_dir,
  target_size=(IM_WIDTH, IM_HEIGHT),
  class_mode = "categorical"
)

but I don't get the same accuracy on both !

epoch 30/30 - loss: 0.3485 - acc: 0.8938 - val_loss: 1.7545 - val_acc: 0.4406

It seems that I am overfitting the training set compared to the validation set.. but they are supposed to be the same ! How is that possible ?

check that there is no dropout or regularizer activated during validation — Tbertin, Aug 29 '18 at 08:24
Possibly of help: [ResNet: 100% accuracy during training, but 33% prediction accuracy with the same data](https://stackoverflow.com/questions/47157526/resnet-100-accuracy-during-training-but-33-prediction-accuracy-with-the-same) — desertnaut, Aug 29 '18 at 13:44

score 1 · Answer 1 · answered Aug 29 '18 at 08:35

1

The training loss is calculated on the fly and only the validation loss is calculated after the epoch is trained. So at the beginning a nearly untrained net will make the training loss worse that it actually is. This effect should vanish in later epochs, since then one epochs mpact on the scoring is not that big anymore.

This behaviour is adressed in keras faq. If you evaluate both at the end of epoch with a self written callback, they should be the same.

answered Aug 29 '18 at 08:35

dennis-w

2,166
1
13
23

1

I don't understand your answer. There should not be a massive difference between calculation before and after the end of the epoch. I should precise that I do not use dropout, and that after 10 epochs the CNN starts clearly to overfit the training set (acc > 0.9) where it is still at ~0.4 on validation set.. – Arcyno Aug 29 '18 at 11:36
Hm you're right I overlooked the fact that you're talking about the 30. epoch. While it's still true, that loss and validation loss should not be exactly the same, this is indeed unexpected behaviour. How are the values in the epochs before? – dennis-w Aug 29 '18 at 11:46
It starts well for about 5 epochs (i.e. both accuracies are similar ~0.5) but it starts diverging just after, getting worse and worse. I am using mobilenet network, so there might be some dropout layers that explain some differences but it should not be that strange... And it would even be the opposite if dropouts layers were the problem – Arcyno Aug 29 '18 at 12:10
I agree on the dropout part. So accuracy of 0.5 does mean the Network does not know anything from the data. Since your training data starts to overfit but not your validation data, the most likely reason would be that your labels get mixed up in either training or validation dataset. – dennis-w Aug 29 '18 at 14:13

score 1 · Answer 2 · answered Jun 06 '19 at 14:03

1

For people reading this after a while : I still don't understand how this issue happened but it helped a lot working on the batchsize (reducing it).

answered Jun 06 '19 at 14:03

Arcyno

4,153
3
34
52

keras CNN : train and validation set are identical but with different accuracy

2 Answers2