Keras callback ModelCheckpoint doesn't save weights

Question

Solution suggested here did not solve my problem this question.

I'm trying to use callbacks ModelCheckpoint and EarlyStopping to save the best weights when early stopping. After the first epoch I get a runtime warning but the codes runs the remaining epochs without errors, but still no file containing weight appears. The warning after the first epoch is the following:

RuntimeWarning: Can save best model only with val_acc available, skipping. 'skipping.' % (self.monitor), RuntimeWarning) RuntimeWarning: Early stopping conditioned on metric val_acc which is not available. Available metrics are: val_loss,val_accuracy,loss,accuracy (self.monitor, ','.join(list(logs.keys()))), RuntimeWarning

I have validation data added to the fit() function so I'm not sure why.

filepath = "weights_best.hdf5"

model.compile(loss="mean_squared_error",
                  metrics=['accuracy'],
                  optimizer=optimizer)
batchSize = 64 
numEpochs = 75
validation_data = (data.x_valid, data.y_valid)

callbackCheckpoint = keras.callbacks.callbacks.ModelCheckpoint(filepath,
                                                                   monitor='val_acc',
                                                                   save_best_only=True,
                                                                   save_weights_only= True,
                                                                   mode='max')

callbackEarlyStop = keras.callbacks.callbacks.EarlyStopping(monitor='val_acc',
                                                                min_delta=0,
                                                                patience=7,
                                                                verbose=0,
                                                                mode='auto')
callbacks = [callbackCheckpoint, callbackEarlyStop]
model.fit(data.x_train, data.y_train, batchSize, numEpochs, callbacks=callbacks,
              validation_data=validation_data)

Any help would be greatly appreciated!

What's the output when you run `fit`? If `val_acc` is there then I'd suggest you change the Keras and Tensorflow version, the code doesn't look like there's any problem. — Natthaphon Hongcharoen, Nov 03 '19 at 16:30
At the end of each epoch it still outputs ```45000/45000 [==============================] - 126s 3ms/step - loss: 0.1793 - accuracy: 0.0993 - val_loss: 0.1806 - val_accuracy: 0.0968```, so yes, ```val_acc``` is present! I'll give it a try. — JustAnotherUser, Nov 03 '19 at 16:41
Nooooooo, that's a `val_accuracy` not `val_acc`. Change that!! — Natthaphon Hongcharoen, Nov 03 '19 at 16:49
`monitor='val_acc'` to `monitor='val_accuracy'` or `metrics=['accuracy']` to `metrics=['acc']` — Natthaphon Hongcharoen, Nov 03 '19 at 17:30
What you're monitoring must be in the printed metrics when you call `fit` — Natthaphon Hongcharoen, Nov 03 '19 at 17:32
Feel free to add this as an answer so I can mark this question as solved! — JustAnotherUser, Nov 05 '19 at 20:30

score 6 · Accepted Answer · answered Nov 06 '19 at 02:40

6

Change monitor='val_acc' to monitor='val_accuracy' or metrics=['accuracy'] to metrics=['acc']

answered Nov 06 '19 at 02:40

Natthaphon Hongcharoen

2,244
1
9
23

score 0 · Answer 2 · answered Mar 07 '20 at 18:00

Yes monitor 'val_accuracy', Just a comment, monitoring and saving the weights based on the highest val_accuracy may not give you the "best" answer. Monitoring 'val_loss' is usually a better answer. val_loss and val_accuracy are calculated in entirely different ways. It is not uncommon to see the val_loss increase but have val_accuracy increase as well or vice versa. In the end you want to use a model with the lowest validation loss. If you do do that make sure to change mode='min' or mode='auto in the checkpoint callback.

Keras callback ModelCheckpoint doesn't save weights

2 Answers2