Keras - training loss vs validation loss

Question

Just for the sake of the argument I am using the same data during training for training and validation, like this:

model.fit_generator(
    generator=train_generator,
    epochs=EPOCHS,
    steps_per_epoch=train_generator.n // BATCH_SIZE,
    validation_data=train_generator,
    validation_steps=train_generator.n // BATCH_SIZE
)

So I would expect that the loss and the accuracy of training and validation at the end of each epoch would be pretty much the same? Still it looks like this:

Epoch 1/150
26/26 [==============================] - 55s 2s/step - loss: 1.5520 - acc: 0.3171 - val_loss: 1.6646 - val_acc: 0.2796
Epoch 2/150
26/26 [==============================] - 46s 2s/step - loss: 1.2924 - acc: 0.4996 - val_loss: 1.5895 - val_acc: 0.3508
Epoch 3/150
26/26 [==============================] - 46s 2s/step - loss: 1.1624 - acc: 0.5873 - val_loss: 1.6197 - val_acc: 0.3262
Epoch 4/150
26/26 [==============================] - 46s 2s/step - loss: 1.0601 - acc: 0.6265 - val_loss: 1.9420 - val_acc: 0.3150
Epoch 5/150
26/26 [==============================] - 46s 2s/step - loss: 0.9790 - acc: 0.6640 - val_loss: 1.9667 - val_acc: 0.2823
Epoch 6/150
26/26 [==============================] - 46s 2s/step - loss: 0.9191 - acc: 0.6951 - val_loss: 1.8594 - val_acc: 0.3342
Epoch 7/150
26/26 [==============================] - 46s 2s/step - loss: 0.8811 - acc: 0.7087 - val_loss: 2.3223 - val_acc: 0.2869
Epoch 8/150
26/26 [==============================] - 46s 2s/step - loss: 0.8148 - acc: 0.7379 - val_loss: 1.9683 - val_acc: 0.3358
Epoch 9/150
26/26 [==============================] - 46s 2s/step - loss: 0.8068 - acc: 0.7307 - val_loss: 2.1053 - val_acc: 0.3312

Why does especially the accuracy differ so much although its from the same data source? Is there something about the way how this is calculated that I am missing?

The generator is created like this:

train_images = keras.preprocessing.image.ImageDataGenerator(
    rescale=1./255
)

train_generator = train_images.flow_from_directory(
    directory="data/superheros/images/train",
    target_size=(299, 299),
    batch_size=BATCH_SIZE,
    shuffle=True
)

Yes, it shuffles the images, but as it iterates over all images also for validation, shouldn't the accuracy at least be close?

So the model looks like this:

inceptionV3 = keras.applications.inception_v3.InceptionV3(include_top=False)

features = inceptionV3.output

net = keras.layers.GlobalAveragePooling2D()(features)
predictions = keras.layers.Dense(units=2, activation="softmax")(net)

for layer in inceptionV3.layers:
    layer.trainable = False

model = keras.Model(inputs=inceptionV3.input, outputs=predictions)
optimizer = keras.optimizers.RMSprop()
model.compile(
    optimizer=optimizer,
    loss="categorical_crossentropy",
     metrics=['accuracy']
)

So no dropout or anything, just the inceptionv3 with a softmax layer on top. I would expect that the accuracy differs a bit, but not in this magnitude.

A lot more detail is needed to answer, like the actual model code, and if the batch size divides the number of samples exactly. Also note that if you use regularization like Dropout or BatchNorm, these work differently on training and testing phases, which explain the difference. — Dr. Snoopy, Oct 29 '18 at 14:56
That looks indeed odd. The accuracy also looks like there is some overfitting happening. What happens if you compute the loss manually? Is it closer to the training or the validation loss? Does the behaviour change if you use different generator-objects pointing to the same folder instead of reusing the same one? — sietschie, Oct 29 '18 at 15:37
@sietschie I tried the approach with two different generators, but it doesnt change anything. — Andreas S, Oct 29 '18 at 16:05
@sietschie Overfitting, yes, but how can you get overfitting when training and validation data are the same? — Andreas S, Oct 29 '18 at 18:03
Possible duplicate of [Keras: Accuracy Drops While Finetuning Inception](https://stackoverflow.com/questions/52282108/keras-accuracy-drops-while-finetuning-inception) — today, Oct 30 '18 at 00:57

score 0 · Answer 1 · answered Oct 29 '18 at 14:40

0

Are you sure your train_generator returns the same data when Keras retrieves training data and validation data, if it's a generator?

The name being generator, I'd expect it not to :)

answered Oct 29 '18 at 14:40

AKX

152,115
15
115
172

Hm, its a folder of images, so it should be - except for the order - the same. – Andreas S Oct 29 '18 at 14:53
Point is, if you're using the same generator object, you don't know if Keras could be reading it in, say, a round-robin order, so every other datum returned is for training, every other for validation. – AKX Oct 29 '18 at 18:35

Keras - training loss vs validation loss

1 Answers1