tf.keras `predict()` gets different results

Question

I was playing around with tf.keras and ran some predict() method on two Model objects with the same weights initialization.

import numpy as np
import tensorflow as tf
from tensorflow.keras.layers import LSTM, Masking, Input, Embedding, Dense
from tensorflow.keras.models import Model

tf.enable_eager_execution()
np.random.seed(10)

X = np.asarray([
    [0, 1, 2, 3, 3],
    [0, 0, 1, 1, 1],
    [0, 0, 0, 1, 1],
])

y = [
    0,
    1,
    1
]

seq_len = X.shape[1]

inp = Input(shape=[seq_len])
emb = Embedding(4, 10, name='embedding')(inp)

x = emb
x = LSTM(5, return_sequences=False, name='lstm')(x)
out = Dense(1, activation='sigmoid', name='out')(x)

model = Model(inputs=inp, outputs=out)
model.summary()

preds = model.predict(X)

inp = Input(shape=[seq_len])
emb = Embedding(4, 10, name='embedding', weights=model.get_layer('embedding').get_weights()[0])(inp)

x = emb
x = LSTM(5, return_sequences=False, weights=model.get_layer('lstm').get_weights()[0])(x)
out = Dense(1, activation='sigmoid', weights=model.get_layer('out').get_weights()[0])(x)

model_2 = Model(inputs=inp, outputs=out)
model_2.summary()

preds_2 = model_2.predict(X)

print(preds, preds_2)

I am not sure why but the results of the two predictions are different. I got these when I ran the print function. You might get something different.

[[0.5027414 ]
 [0.5019673 ]
 [0.50134844]] [[0.5007331]
 [0.5002397]
 [0.4996575]]

I am trying to understand how keras works. Any explanation would be appreciated. Thank you.

NOTE: THERE IS NO LEARNING INVOLVED HERE. I don't get the idea where the randomness comes from.

@Justice_Lords I didnt call any model.fit(). I am just initializing a model with the same weights as some other models and predict on `X` — Hardian Lawi, Mar 15 '19 at 03:21
I don't understand why this is marked as duplicate to https://stackoverflow.com/questions/50659482/why-cant-i-get-reproducible-results-in-keras-even-though-i-set-the-random-seeds I didn't train the model or anything, so I don't get the idea where the randomness comes from. — Hardian Lawi, Mar 15 '19 at 03:23

score 0 · Answer 1 · answered Jul 27 '19 at 22:01

0

Try to change the optimizer from adam to SGD or something else. I noticed that with the same model I used to get different results and it fixed the problem. Also, take a look at the here to fix the initial weights. By the way, I don't know why and how the optimizer can affect the results in the test time with the same model.

answered Jul 27 '19 at 22:01

The code in the question does not do any training, hence no optimizer is used at all. The initialization of the second model is also done with the same weights as the first model. This post does not really answer the question about where the difference of outcomes come from. It would be better that you can read the original post, others' answers and comments more carefully before posting an answer. – GZ0 Jul 27 '19 at 22:54

score 0 · Answer 2 · edited May 26 '20 at 15:45

It is that you are not copying all the weights. I have no idea why your call mechanically works but it is really easy to see you are not by examining the get_weights without the [0] indexing.

e.g.these are not copied:

model.get_layer('lstm').get_weights()[1]


array([[ 0.11243069, -0.1028666 ,  0.01080172, -0.07471965,  0.05566487,
        -0.12818974,  0.34882438, -0.17163819, -0.21306667,  0.5386005 ,
        -0.03643916,  0.03835883, -0.31128728,  0.04882491, -0.05503649,
        -0.22660127, -0.4683674 , -0.00415642, -0.29038426, -0.06893865],
       [-0.5117522 ,  0.01057898, -0.23182054,  0.03220385,  0.21614116,
         0.0732751 , -0.30829042,  0.06233712, -0.54017985, -0.1026137 ,
        -0.18011908,  0.15880923, -0.21900705, -0.11910527, -0.03808065,
         0.07623457, -0.13157862, -0.18740109,  0.06135096, -0.21589288],
       [-0.2295578 , -0.12452635, -0.08739456, -0.1880849 ,  0.2220488 ,
        -0.14575425,  0.32249492,  0.05235165, -0.09479579,  0.2496742 ,
         0.10411342, -0.0263749 ,  0.33186644, -0.1838699 ,  0.28964192,
        -0.2414586 ,  0.41612682,  0.13791762,  0.13942356, -0.36176005],
       [-0.14428475, -0.02090888,  0.27968913,  0.09452424,  0.1291543 ,
        -0.43372717, -0.11366601,  0.37842247,  0.3320751 ,  0.21959782,
        -0.4242381 ,  0.02412989, -0.24809352,  0.2508208 , -0.06223384,
         0.08648364,  0.17311276, -0.05988384,  0.02276517, -0.1473657 ],
       [ 0.28600952, -0.37206012,  0.21376705, -0.16566195,  0.0833357 ,
        -0.00887177,  0.01394618,  0.5345957 , -0.25116244, -0.17159337,
         0.096329  , -0.32286254,  0.02044407, -0.1393016 , -0.0767666 ,
         0.1505355 , -0.28456056,  0.16909163,  0.16806729, -0.14622769]],
      dtype=float32)

but also if you name the lstm layer in model 2 you can see there are not equal parts of the weights.

model_2.get_layer("lstm").get_weights()[1] - model.get_layer("lstm").get_weights()[1]

score 0 · Answer 3 · answered Jun 10 '22 at 08:07

Perhaps, setting numpy seed is not enough to make the operations and weights deterministic. Tensorflow documentation suggests that to have deterministic weights, you should rather run

tf.keras.utils.set_random_seed(1)
tf.config.experimental.enable_op_determinism()

https://www.tensorflow.org/api_docs/python/tf/config/experimental/enable_op_determinism#:~:text=Configures%20TensorFlow%20ops%20to%20run%20deterministically.&text=When%20op%20determinism%20is%20enabled,is%20useful%20for%20debugging%20models.

Could you check if it helps? (your code seems to be written in version 1 of TF, so it does not run on my v2 setup without adaptation)

Jessica Chambers · Answer 4 · 2019-07-29T08:08:45.603

The thing about machine learning is that it doesn't always learn quite the same way. It involves lots of probabilities, so on a larger scale the results will tend to converge towards one value, but individual runs can and will give varying results.

More info here

It is absolutely normal that the many runs with the same input data give different output. It is mainly due to the internal stochasticity of such machine learning techniques (example: ANN, Decision Trees building algorithms, etc.).

- Nabil Belgasmi, Université de la Manouba

There is not a specific method or technique. The results and evaluation of the performance depends on several factors: the data type, parameters of induction function, training set (supervised), etc. What is important is to compare the results of using metric measurements such as recall, precision, F_measure, ROC curves or other graphical methods.

- Jésus Antonio Motta Laval University

EDIT The predict() function takes an array of one or more data instances.

The example below demonstrates how to make regression predictions on multiple data instances with an unknown expected outcome.

# example of making predictions for a regression problem
from keras.models import Sequential
from keras.layers import Dense
from sklearn.datasets import make_regression
from sklearn.preprocessing import MinMaxScaler
# generate regression dataset
X, y = make_regression(n_samples=100, n_features=2, noise=0.1, random_state=1)
scalarX, scalarY = MinMaxScaler(), MinMaxScaler()
scalarX.fit(X)
scalarY.fit(y.reshape(100,1))
X = scalarX.transform(X)
y = scalarY.transform(y.reshape(100,1))
# define and fit the final model
model = Sequential()
model.add(Dense(4, input_dim=2, activation='relu'))
model.add(Dense(4, activation='relu'))
model.add(Dense(1, activation='linear'))
model.compile(loss='mse', optimizer='adam')
model.fit(X, y, epochs=1000, verbose=0)
# new instances where we do not know the answer
Xnew, a = make_regression(n_samples=3, n_features=2, noise=0.1, random_state=1)
Xnew = scalarX.transform(Xnew)
# make a prediction
ynew = model.predict(Xnew)
# show the inputs and predicted outputs
for i in range(len(Xnew)):
    print("X=%s, Predicted=%s" % (Xnew[i], ynew[i]))

Running the example makes multiple predictions, then prints the inputs and predictions side by side for review.

X=[0.29466096 0.30317302], Predicted=[0.17097184]
X=[0.39445118 0.79390858], Predicted=[0.7475489]
X=[0.02884127 0.6208843 ], Predicted=[0.43370453]

SOURCE

Disclaimer: The predict() function itself is slightly random (probabilistic)

What would you expect in an actual answer? OP asked for an explanation. — Jessica Chambers, Mar 14 '19 at 09:43
@JessicaChambers I didn't retrain any model. I just simply initialized some model with the weights of another model and predict on the same `X`. There is no learning involved here. — Hardian Lawi, Mar 15 '19 at 03:26
@HardianLawi The `predict()` function itself is slightly random (probabilistic) and is a form of machine learning. Check out section 3. Regression Predictions on this page https://machinelearningmastery.com/how-to-make-classification-and-regression-predictions-for-deep-learning-models-in-keras/ — Jessica Chambers, Mar 15 '19 at 10:31
@JessicaChambers Why is the `predict()` function stochastic? Plus, you don't get my point. I didn't train any model. — Hardian Lawi, Mar 16 '19 at 11:40
@JessicaChambers, I think you'd improve your answer if you added the comment "The predict() function itself is slightly random..." to the main body of this answer. — , Jul 27 '19 at 22:08
I think you've confused *training* with *prediction*. Your source refers to predict preceded by a (stochastic) training stage. As the OP notes the **weights are fixed**, so there is no training step, or any ML for that matter, and so the results should be the same each time. — c z, Oct 08 '20 at 11:40

tf.keras `predict()` gets different results

4 Answers4