Trouble with EarlyStopping in tf.keras

Question

I am training my first transferred learning model(yay!) and I am having trouble getting the model to stop training when the validation loss hasn't changed by more than 0.1 in more than 3 epochs.

Here is the relevant block of code

early_stopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3, min_delta = 0.1)

model.compile(optimizer='adam',
          loss='sparse_categorical_crossentropy',
          metrics=['accuracy'],
          callbacks=[early_stopping])

EPOCHS = 100

history = model.fit(training_batches,
                epochs=EPOCHS,
                validation_data=validation_batches)

And here are some of the logs:

    Epoch 32/100
    155/155 [==============================] - 21s 134ms/step - loss: 0.0042 - accuracy: 0.9998 - val_loss: 0.3407 - val_accuracy: 0.9012
    Epoch 33/100
    155/155 [==============================] - 21s 133ms/step - loss: 0.0040 - accuracy: 0.9998 - val_loss: 0.3443 - val_accuracy: 0.9000
    Epoch 34/100
    155/155 [==============================] - 21s 134ms/step - loss: 0.0037 - accuracy: 0.9998 - val_loss: 0.3393 - val_accuracy: 0.9019
    Epoch 35/100
    155/155 [==============================] - 21s 135ms/step - loss: 0.0031 - accuracy: 1.0000 - val_loss: 0.3396 - val_accuracy: 0.9000
    Epoch 36/100
    155/155 [==============================] - 21s 134ms/step - loss: 0.0028 - accuracy: 1.0000 - val_loss: 0.3390 - val_accuracy: 0.9000
    Epoch 37/100
    155/155 [==============================] - 21s 133ms/step - loss: 0.0026 - accuracy: 1.0000 - val_loss: 0.3386 - val_accuracy: 0.9025
    Epoch 38/100
    155/155 [==============================] - 21s 133ms/step - loss: 0.0024 - accuracy: 1.0000 - val_loss: 0.3386 - val_accuracy: 0.8994
    Epoch 39/100
    155/155 [==============================] - 21s 133ms/step - loss: 0.0022 - accuracy: 1.0000 - val_loss: 0.3386 - val_accuracy: 0.9019

Questions:

• Why didn't training stop at Epoch 37 when I have the EarlyStopping callback set to monitor val_loss?
• Can I do more complex EarlyStopping callbacks? Something along the lines of "If val_accuracy > 0.90 && val_loss hasn't changed by more than 0.1 in 3 Epochs". If I can, can I get a link to a tutorial?

Answer 1

EDIT

It doesn't work because you placed the callback parameter in the wrong method call. (and in fact, I received an invalid argument error when fitting the model with callbacks passed to compile. Thus I'm not sure why your model compiled without problem.)

It should be inside your fit method, as shown below. Note that it is recommended to set verbose = 1 in your early stopping configuration, so that it will prints out the early stopping log.

early_stopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3, min_delta = 0.1, verbose = 1)

model.compile(optimizer='adam',
          loss='sparse_categorical_crossentropy',
          metrics=['accuracy'])

EPOCHS = 100

history = model.fit(training_batches,
                epochs=EPOCHS,
                callbacks=[early_stopping], # call back should be here!
                validation_data=validation_batches)

On your second question, custom callback is possible, you can refer to the example from documentation here. Basically you got to define your early stopping logic in on_epoch_end.

On a side note, I think you shouldn't be early stopping on multiple metrics, pick one that matters (i.e. the metric that you are optimizing - val_accuracy) and just monitor that. There are even sources which discourage early stopping & instead, treat epoch as a tunable hyperparameter. See this discussion thread on Reddit which I found useful.

Answer 2

Decrease patience=3 to less e. g. 1 or 2 and see what happens.

It tells Keras how hard you want to try. patience= small number will tell Keras to stop the training early. On the other hand, if you use a big number it will tell Keras to wait until a significant amount of accuracy is achieved

patience: number of epochs that produced the monitored quantity with no improvement after which training will be stopped. Validation quantities may not be produced for every epoch, if the validation frequency (model.fit(validation_freq=5)) is greater than one