2
  1. I set both numpy and tensorflow random seeds as suggested
  2. Generate some data - this part is reproducible, gives same results always
  3. Create a simple network and make a prediction (without training, just with random weights) - prediction is different every time
import numpy as np
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras import Sequential, optimizers
import tensorflow as tf

np.random.seed(32)
tf.set_random_seed(33)

random_data = np.random.rand(10, 2048)
print(random_data[:,0])

def make_classifier():
    model = Sequential()
    model.add(Dense(1024, activation='relu', input_dim=2048))
    model.add(Dropout(0.5))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(optimizer=optimizers.RMSprop(lr=2e-4),
              loss='binary_crossentropy')
    return model
model = make_classifier()
# model.summary()
model.predict(random_data)

When I re-run the whole cell, the print statement always outputs [0.85888927 0.23846818 0.17757634 0.07244977 0.71119893 0.09223853 0.86074647 0.31838194 0.7568638 0.38197083]. However the prediction is different every time:

array([[0.5825965 ],
       [0.8677979 ],
       [0.70151913],
       [0.64572096],
       [0.78101623],
       [0.76483005],
       [0.7946336 ],
       [0.6281612 ],
       [0.8208673 ],
       [0.8273002 ]], dtype=float32)

array([[0.51012236],
       [0.6562015 ],
       [0.5593666 ],
       [0.686155  ],
       [0.6488372 ],
       [0.5966359 ],
       [0.6236731 ],
       [0.58099884],
       [0.68447435],
       [0.58886844]], dtype=float32)

And so on.

  1. How do I get reproducible prediction for a just-initialized network?
  2. When does the weights initialization happen exactly? when I compile the model or?..
Nicolas Gervais
  • 33,817
  • 13
  • 115
  • 143
Dmitry Toda
  • 23
  • 4
  • Does this answer your question? [How to get reproducible results in keras](https://stackoverflow.com/questions/32419510/how-to-get-reproducible-results-in-keras) – Sycorax Jan 19 '21 at 15:27

3 Answers3

2

I've been struggling a lot with this and turns out there are quite a few points that have to be set in order to achieve complete consistency for every case:

First of all, make sure the data (and the order of the data) that you feed to your model is consistent. Then, for the model weights initialization:

1)numpy random seed

import numpy as np
np.seed(1)

2)tensor flow random seed

import tensorflow as tf
tf.set_random_seed(2)

3)python random seed

import random
random.seed(3)

Additionally to that, you have to set two (if you have multiprocessing capabilities) arguments to model.fit. These ones are not often mentioned on the answers I've seen around:

model.fit(..., shuffle=False, use_multiprocessing=False)

Only then I have achieved complete consistency in training runs.

Hope that helps people!

Lucas Azevedo
  • 1,867
  • 22
  • 39
1

tf.keras.initializers objects have a seed argument for reproducible initialization.

import tensorflow as tf
import numpy as np

initializer = tf.keras.initializers.GlorotUniform(seed=42)

for _ in range(10):
    print(np.round(initializer((4,)), 3))

[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]
[-0.377 -0.003  0.373 -0.831]

In a Keras layer, you can use it like this:

tf.keras.layers.Dense(1024, 
                      activation='relu', 
                      input_dim=2048,
                      kernel_initializer=tf.keras.initializers.GlorotUniform(seed=42))
Nicolas Gervais
  • 33,817
  • 13
  • 115
  • 143
  • Thank you! This helped, although I had to write it like glorot_uniform: ```Dense(1024, activation='relu', input_dim=2048, kernel_initializer=tf.keras.initializers.glorot_uniform(seed=42))``` – Dmitry Toda Jan 19 '21 at 18:38
  • I'm using a neural network of the form `model = Sequential(); model.add(LSTM(nodes, activation='relu', kernel_initializer=tf.keras.initializers.GlorotUniform(seed=1))); ...` but at every execution I was getting different results. Somewhere else I read about `tf.random.set_seed(0)`, so I added it before the second line and now it is working correctly, i.e. at each execution the list `model_history.history['loss']` contains always the same values. – sound wave May 06 '22 at 07:27
0

I think a better practice is

tf.keras.utils.set_random_seed(
    seed
)

ref: https://www.tensorflow.org/api_docs/python/tf/keras/utils/set_random_seed

Litchy
  • 623
  • 7
  • 23