0

I am training a neural network in Keras but when new data comes and I try to retrain it, the loss in the epochs it's as high as the first time I trained my model. 

 checkpoint = ModelCheckpoint('my_model.h5', monitor='loss', verbose=1, save_best_only=True, mode='min')
 callbacks_list = [checkpoint]
 model.fit(X_train,y_train, batch_size = batch_size, epochs = epochs, callback = callbacks_list)
 new_model = load_model('my_model.h5')

As suggested here Keras: How to save model and continue training? I tried to predict the same data both in model and new_model and measure the differences with: 

assert_allclose(model.predict(x_train),
            new_model.predict(x_train),
            1e-5)

In fact, I got the Assertion Error and I did even with tol = 1e-2 so that makes me think my model is not loading as it should. Anyone has a clue of why is this happening?

Marisa
  • 1,135
  • 3
  • 20
  • 33
  • I assume the function `load_model()` is not correct when loading from a model checkpoint. It only stores the weights of your model. Try to load the weights instead with `load_weights()`. [Example](https://machinelearningmastery.com/check-point-deep-learning-models-keras/) – p13rr0m Oct 09 '18 at 15:11
  • As far as I know if you don't put ``save_weights_only = True`` the whole model will be saved, at least that's what Keras documentation says. – Marisa Oct 09 '18 at 17:51

1 Answers1

1

ModelCheckpoint saves the model weighs that has had less loss in training.

Your model has saved the last epoch weighs.

If the last epoch of your model is not the one that has had less loss, the weights of the saved model (new_model) do not match those of the original model and the prediction is not the same.

pfRodenas
  • 326
  • 1
  • 2
  • 12
  • Ok, that makes sense. However, when I retrain my model it seems that even if the loss is higher in the retrain the models will be updated, that shouldn't be happening, right? Any clue of why it happens? – Marisa Oct 09 '18 at 18:03
  • If you want save the last epoch ,not call callbacks in your model.fit `model.fit(X_train,y_train, batch_size = batch_size, epochs = epochs)`, instead use `model.save('my_model.h5')` – pfRodenas Oct 09 '18 at 18:07
  • Sorry, I might have not explained myself. I want to save the best epoch and with that one retrain my model using epochs too, but I don't want to update the previous one that I saved if the loss hasn't improved. – Marisa Oct 09 '18 at 18:11
  • I don't know if that is possible. You can do manually, changing the name of the saved model like my_model1.h5, my_model2.h5 ... Then check the best model acc by `result = model.evaluate(x=x_train,y=y_train)` and `print("{0}: {1:.2%}".format(model.metrics_names[1], result[1]))` – pfRodenas Oct 09 '18 at 18:32