Keras: MLP on CPU Reproducibility

Question

I'm building and testing a simple MLP model, but am running into an issue with the Keras reproducibility for my results. I am trying to set up my neural network so that the prediction outputs won't change when I run the network.

I have already followed the Keras guide online as well as this post (Reproducible results using Keras with TensorFlow backend). I am running Keras on my local machine with Tensorflow backend and the following versions:

tensorflow 2.0.0-alpha0, keras 2.2.4-tf, numpy 1.16.0

import os  
os.environ['PYTHONHASHSEED']=str(0)

import random
random.seed(0)

from numpy.random import seed
seed(1)
import tensorflow as tf
tf.compat.v1.set_random_seed(2)

from keras import backend as K
session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)
K.set_session(sess)

import numpy as np
from tensorflow.python.keras.layers import Dropout, BatchNormalization
from tensorflow.python.keras.optimizers import Adam


class Machine_Learning_Classifier_Keras(object):    
    @classmethod
    def _get_classifier(cls, n_input_features=None, **params):
        KerasClassifier = tf.keras.wrappers.scikit_learn.KerasClassifier
        Dense = tf.keras.layers.Dense
        Sequential = tf.keras.models.Sequential

        sk_params = {"epochs": 200, "batch_size": 128, "shuffle": False}

        def create_model(optimizer='adam', init='he_normal'):
            # create model
            model = Sequential()
            model.add(BatchNormalization())
            model.add(Dropout(0.2))
            model.add(Dense(500, input_dim=4, kernel_initializer=init, activation='relu'))
            model.add(BatchNormalization())
            model.add(Dropout(0.2))
            model.add(Dense(250, kernel_initializer=init, activation='relu'))
            model.add(BatchNormalization())
            model.add(Dropout(0.2))
            model.add(Dense(500, kernel_initializer=init, activation='relu'))
            model.add(Dense(1, kernel_initializer=init, activation='sigmoid'))
            # Compile model
            model.compile(loss='binary_crossentropy', optimizer=Adam(lr=3e-3, decay=0.85), metrics=['accuracy'])
            return model

        return KerasClassifier(build_fn=create_model, **sk_params)

if __name__ == "__main__":
    X = np.asarray([[0.0, 0.0], [1.0, 1.0], [2.0, 2.5], [1.5, 1.6]])
    y = np.asarray([0, 0, 1, 1])

    nn = Machine_Learning_Classifier_Keras._get_classifier()
    nn.fit(X, y, sample_weight=np.asarray([0, 0, 1, 1]))

    values = np.asarray([[0.5, 0.5], [0.6, 0.5], [0.8, 1.0], [0.5, 0.5], [0.5, 0.5], [0.5, 0.5], [0.5, 0.5], [0.5, 0.5]])

    probas = nn.predict_proba(values)
    print(probas)

I would expect my outputs for the predict_proba values to stay the same between runs; however, I am getting the following for two successive runs (results will vary):

    Run 1:
    [[0.9439231  0.05607685]
     [0.91351616 0.08648387]
     [0.06378722 0.9362128 ]
     [0.9439231  0.05607685]
     [0.9439231  0.05607685]
     [0.9439231  0.05607685]
     [0.94392323 0.05607677]
     [0.94392323 0.05607677]]

    Run 2:
    [[0.94391584 0.05608419]
     [0.91350436 0.08649567]
     [0.06378281 0.9362172 ]
     [0.94391584 0.05608419]
     [0.94391584 0.05608419]
     [0.94391584 0.05608419]
     [0.94391584 0.05608416]
     [0.94391584 0.05608416]]

John Doerian · Answer 1 · 2019-06-20T13:50:28.977

1

Ended up figuring out what the issue is, but not sure how to resolve -- it has something to do with the first BatchNormalization() layer, which is supposed to standardize the inputs. If you remove that layer the results are entirely reproducible, but something in the BatchNormalization() implementation is leading to non-reproducible behavior

edited Jun 20 '19 at 13:50

answered Jun 19 '19 at 17:42

John Doerian

51
5

score 0 · Answer 2 · answered Jun 14 '19 at 09:03

0

If you run the mentioned code twice then it will show the behavior that you have just described. Cause every-time the model is being trained and its not necessary that it may lead to same local minimum every-time. However if you train your model only once and save the weights and use those weights to predict the output then you will get same results every-time for same data.

answered Jun 14 '19 at 09:03

Vaibhav gusain

406
3
9

If I am training multiple models in succession (for example on different periods of time in a time series), would I just save the various weights and then reload them using `model.load_weights(...)` after calling `model.compile(...)` ? – John Doerian Jun 14 '19 at 12:56
yes, if you do training again you will get different results. – Vaibhav gusain Jun 15 '19 at 06:24

Keras: MLP on CPU Reproducibility

2 Answers2