3

I am training some model via keras with tensorflow backend.

When I call predict right after training on the same object it works fine and gives different values for different inputs. But when I save the model to a file, then load it from another python session, predict always returns the same value for different inputs.

I use ModelCheckpoint for saving the model, then load_model for loading. I have also tried to save and load architecture separately to a json file with to_json and model_from_json functions. Sample code:

Saving part

with open("model.json", "w") as textFile:
   print(model.to_json(), file = textFile)

model.fit(X_train, y_train, epochs=iterationCount, batch_size=64, validation_split=0.2, callbacks = [ModelCheckpoint(filepath='model.h5', verbose=0, save_best_only=True)])

Loading part

with open('model.json') as json_file:
    model = model_from_json(json_file.read())

model.load_weights('model.h5')

Any ideas to solve this? Is there anything that I am missing?

nabroyan
  • 3,225
  • 6
  • 37
  • 56
  • Could you disable the `save_best_only` option and check if anything changes. – Abhai Kollara Jan 21 '18 at 19:40
  • 3
    How about data preprocessing? Is it the same? If data is not scaled it's very often that you'll obtain the same prediction for all examples. – Marcin Możejko Jan 21 '18 at 20:08
  • @MarcinMożejko Yes, for preprocessing I use scikit learn's `StandardScaler()`. I save and load the scaler in a file with `joblib.dump()` and `joblib.load()` functions. – nabroyan Jun 13 '18 at 21:10
  • Any update on this? I am having the same issue and I don't know how to resolve it. – M. Barbieri Jan 24 '19 at 00:25
  • 1
    @MarcinMożejko was right, there was a bug in data preprocessing code. There were very big numbers in my data and due to that bug they have not been cleared/scaled, which caused to behavior described in the question. – nabroyan Jan 25 '19 at 07:38

3 Answers3

2

Unfortunately many people (like me) have been complaining on a keras bug that affects the save_weights and load_weights functions.

I ended up avoiding to use these function. If I want to load/store my model I just do the following:

from keras.models import load_model

trained_model.save(file_path)
loaded_model = load_model(file_path)

The save functions saves your weights, but also your network structure and the state of the optimizer (it makes sure that you can keep training model from exactly where you paused).

rickyalbert
  • 2,552
  • 4
  • 21
  • 31
2

I had the same issue, I solved by setting a fixed seed to tensorflow, numpy and python.

import tensorflow as tf
import numpy as np
import random as python_random
tf.random.set_random_seed(42)
np.random.seed(42)
python_random.seed(42)

Be careful! Different versions of tensorflow may required a different way to set a seed!

Raphael Mazzine
  • 61
  • 4
1

keras model save/load has two ways (two ways I feel comfortable to use). Saving both model structure and weights model.save(file_path) and only save model weights model.save_weights(file_path). So I save/load model like following:

from keras.models import load_model
trained_model.save(file_path)
loaded_model = load_model(file_path)

trained_model.save_weights(file_path)
# We have to define model first before loading weights
trained_model.load_weights(file_path)

Here, load_weights(file_path, by_name=True) allows you to load weights into a different structure with some layer names in common.

Xiao Wang
  • 11
  • 1