5

I am training Keras models, saving them with model.save() and than later loading them and resuming training.

I would like to plot after each training the whole training history, but model.fit_generator() only returns the history of the last session of training.

I can save the history of the initial session and update it myself, but I wonder if there is a standard way in Keras of managing the training history.

history1 = model.fit_generator(my_gen)
plot_history(history1)
model.save('my_model.h5')

# Some days afterwards...

model = load_model('my_model.h5')
history2 = model.fit_generator(my_gen)

# here I would like to reconstruct the full_training history
# including the info from history1 and history2
full_history = ???
Jsevillamol
  • 2,425
  • 2
  • 23
  • 46

3 Answers3

1

Let's say this line

print(history.history.keys())

produces the following output:

['acc', 'loss', 'val_acc', 'val_loss']

Based on the assumption that the loaded model should have the same performance as saved model, you can try something like concatenating histories. For example, concatenate new accuracy history on the loaded accuracy history of the loaded model.

It should start from the same point in the plotting space where the loaded model ended (maybe you will have to add (+) epochs of the previously trained model for the plot so that new values of accuracy don't start from epoch 0, but loaded models' last epoch).

I hope that you understand my idea and that it will help you :)

Novak
  • 2,143
  • 1
  • 12
  • 22
1

Turns out there is no standard way of doing this in Keras yet AFAIK.

See this issue for context.

Jsevillamol
  • 2,425
  • 2
  • 23
  • 46
1

Use numpy to concatenate the specific history keys that you are interested in.

For example, let's say these are your two training runs:

history1 = model.fit_generator(my_gen)
history2 = model.fit_generator(my_gen)

You can view the dictionary keys, which will be labeled the same for each run by:

print(history1.history.keys())

This will print the an output like:

dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])

Then you can use numpy concatenate. For example:

import numpy as np
combo_accuracy=np.concatenate((history1.history['accuracy'],history2.history['accuracy']),axis=0)
combo_val_accuracy=np.concatenate((history1.history['val_accuracy'],history2.history['val_accuracy']),axis=0)

You can then plot the concatenated history arrays with matplotlib:

import matplotlib.pyplot as plt
plt.plot(combo_acc, 'orange', label='Training accuracy')
plt.plot(combo_val_acc, 'blue', label='Validation accuracy')
plt.legend()
Paras Lakhani
  • 26
  • 1