Keras model predicts different results using the same input

Question

I built a Keras sequential model on the simple dataset. I am able to train the model, however every time I try to get a prediction on the same input I get different values. Anyone knows why? I read through different Stackoverflow here (Why the exactly identical keras model predict different results for the same input data in the same env, Keras saved model predicting different values on different session, different prediction after load a model in keras), but couldn't find the answer. I tried to set the Tensorflow seed and still getting different results. Here is my code

from pandas import concat
from pandas import DataFrame
# create sequence
length = 10
sequence = [i/float(length) for i in range(length)]
# create X/y pairs
df = DataFrame(sequence)
df = concat([df, df.shift(1)], axis=1)
df.dropna(inplace=True)
print(df)
# convert to LSTM friendly format
values = df.values
X, y = values[:, 0], values[:, 1]
X = X.reshape(len(X), 1, 1)
print(X.shape, y.shape)

output is:

     0    0
1  0.1  0.0
2  0.2  0.1
3  0.3  0.2
4  0.4  0.3
5  0.5  0.4
6  0.6  0.5
7  0.7  0.6
8  0.8  0.7
9  0.9  0.8
(9, 1, 1) (9,)

Then start building the model

#configure network
from tensorflow import keras
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
tf.random.set_seed(1337)

n_batch = len(X)
n_neurons = 10
#design network
model = Sequential()
model.add(LSTM(n_neurons, batch_input_shape=(n_batch, X.shape[1], X.shape[2]), stateful=True))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(X,y,epochs=2,batch_size=n_batch,verbose=1,shuffle=False)

Now every time I run the following code to get the prediction I get different results as you can see here

model.predict(X)
********output**************
 array([[0.03817442],
       [0.07164046],
       [0.10493257],
       [0.13797525],
       [0.17069395],
       [0.20301574],
       [0.23486984],
       [0.26618803],
       [0.29690543]], dtype=float32)

model.predict(X)
********output**************
array([[0.04415776],
       [0.08242793],
       [0.12048437],
       [0.15823033],
       [0.19556962],
       [0.2324073 ],
       [0.26865062],
       [0.3042098 ],
       [0.33899906]], dtype=float32)

I am not sure, but I think it setting stateful to True will save the last state for each sample and use it as initial state for next sample. But I thought that is for building the model only, am I wrong? here is the explanation by Tensorflow docs. https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM — Shahin Shirazi, Jan 31 '23 at 23:30
correct, but it is not only for training, also for prediction — Dr. Snoopy, Jan 31 '23 at 23:31
Thank you so much, you are correct. I turned that to False and started working. Appreciate it. — Shahin Shirazi, Jan 31 '23 at 23:33

score 2 · Answer 1 · answered Jan 31 '23 at 23:28

2

I think this library and the documentation attached to it will be interesting for your work.

Based on the above library, in a recent work I had with the Keras, I was starting the code as follows:

import os
import numpy as np
from numpy.random import seed
seed(42)
rng = np.random.RandomState(42)
import tensorflow
tensorflow.random.set_seed(42)
os.environ['TF_DETERMINISTIC_OPS'] = '1'

There seemed to be a good deal of determinism in the results, and it was good enough for what I was working on at the time.

answered Jan 31 '23 at 23:28

Higo Felipe Silva Pires

63
10

No, this is just an incorrect guess. – Dr. Snoopy Jan 31 '23 at 23:29
@Dr.Snoopy Is it possible that these models can behave deterministically? If yes, how to do this programmatically? – Higo Felipe Silva Pires Jan 31 '23 at 23:33

score 2 · Accepted Answer · answered Jan 31 '23 at 23:39

2

The problem is setting stateful=True in your LSTM layer, as this keeps the state between predict calls, so each prediction depends on previous predictions.

So as a solution, set stateful=False.

answered Jan 31 '23 at 23:39

Dr. Snoopy

55,122
7
121
140

score 0 · Answer 3 · answered Jan 31 '23 at 23:38

Based on @Dr.Snoopy the problem was setting stateful = True. Setting it to False fixed the issue. "Boolean (default False). If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch." and my misunderstanding was that this only applies to training.

Thanks to @Dr.Snoopy for pointing that out.

Keras model predicts different results using the same input

3 Answers3