0

I have this tensorflow code for MLP. This works just fine but the result is not reproducible. I have added seeds but still can't reproduce the result. Everytime I run the program, new result is generated.

import tensorflow as tf
tf.set_random_seed(1337)
import numpy as np
np.random.seed(1337)
import random
random.seed(1337)

# Train-test split
        data_train, data_test, labels_train, labels_test = train_test_split(data, labels_, test_size=TEST_SIZE,                                                                           random_state=RANDOM_STATE)

        def DeepMLPClassifier(_X, _weights, _biases, dropout_keep_prob):
            layer1 = tf.nn.dropout(tf.nn.tanh(tf.add(tf.matmul(_X, _weights['h1']), _biases['b1'])), dropout_keep_prob)
            layer2 = tf.nn.dropout(tf.nn.tanh(tf.add(tf.matmul(layer1, _weights['h2']), _biases['b2'])), dropout_keep_prob)
            layer3 = tf.nn.dropout(tf.nn.tanh(tf.add(tf.matmul(layer2, _weights['h3']), _biases['b3'])), dropout_keep_prob)
            layer4 = tf.nn.dropout(tf.nn.tanh(tf.add(tf.matmul(layer3, _weights['h4']), _biases['b4'])), dropout_keep_prob)
            out = ACTIVATION_FUNCTION_OUT(tf.add(tf.matmul(layer4, _weights['out']), _biases['out']))
            return out


        # Here are the dictionary of weights and biases of each layer
        weights = {
            'h1': tf.Variable(tf.random_normal([n_input, n_hidden_1], stddev=STDDEV, seed=1337)),
            'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2], stddev=STDDEV, seed=1337)),
            'h3': tf.Variable(tf.random_normal([n_hidden_2, n_hidden_3], stddev=STDDEV, seed=1337)),
            'h4': tf.Variable(tf.random_normal([n_hidden_3, n_hidden_4], stddev=STDDEV, seed=1337)),
            'out': tf.Variable(tf.random_normal([n_hidden_4, n_classes], stddev=STDDEV, seed=1337)),
        }

        biases = {
            'b1': tf.Variable(tf.random_normal([n_hidden_1], seed=1337)),
            'b2': tf.Variable(tf.random_normal([n_hidden_2], seed=1337)),
            'b3': tf.Variable(tf.random_normal([n_hidden_3], seed=1337)),
            'b4': tf.Variable(tf.random_normal([n_hidden_4], seed=1337)),
            'out': tf.Variable(tf.random_normal([n_classes], seed=1337))
        }

        # Build model
        pred = DeepMLPClassifier(X, weights, biases, dropout_keep_prob)

        # Loss and optimizer
        cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))  # softmax loss
        optimizer = tf.train.AdamOptimizer(learning_rate=LEARNING_RATE).minimize(cost)

        # Accuracy
        correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
        accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

        print("Deep MLP networks has been built successfully...")
        print("Starting training...")
        # ------------------------------------------------------------------------------
        # Initialize variables
        init_op = tf.global_variables_initializer()
        # Launch session
        sess = tf.Session()
        sess.run(init_op)
        # tf.set_random_seed(1)


        acc_list = []
        cost_list = []
        i_data = []

        # Training loop
        for epoch in range(TRAINING_EPOCHS):
            avg_cost = 0.0
            total_batch = int(data_train.shape[0] / BATCH_SIZE)
            # Loop over all batches
            for i in range(total_batch):
                randidx = np.random.randint(int(TRAIN_SIZE), size=BATCH_SIZE)
                batch_xs = data_train[randidx, :]
                batch_ys = labels_train[randidx, :]
                # Fit using batched data
                sess.run(optimizer, feed_dict={X: batch_xs, y: batch_ys, dropout_keep_prob: 0.9})
                # Calculate average cost
                avg_cost += sess.run(cost, feed_dict={X: batch_xs, y: batch_ys, dropout_keep_prob: 1.}) / total_batch
                # Display progress
            # if epoch % DISPLAY_STEP == 0:
            i_data.append(epoch + 1)
            cost_list.append(avg_cost)
            print("\n Epoch:%3d/%3d, cost:%.9f" % (epoch, TRAINING_EPOCHS, avg_cost))
            train_acc = sess.run(accuracy, feed_dict={X: batch_xs, y: batch_ys, dropout_keep_prob: 1.})
            acc_list.append(train_acc)
            print("Training accuracy: %.3f" % (train_acc))

What am I missing here? How can I reproduce the results? I have even added same seeds to weights and biases. I have tried without seeds as well. But nothings seems to be working.

user1670773
  • 967
  • 4
  • 12
  • 23
  • 3
    can't it be due to the multiprocessing in TF, that causes your dropouts to be called in different orders each time ? – gdelab Jan 16 '18 at 13:31
  • you might be correct. I just checked, in my another laptop it always produce same results. This is always produce different results. What can I do? This one is new with core-i7, other one i3 – user1670773 Jan 16 '18 at 15:47
  • I don't have the problem myself on a smaller test case (on i7), so I don't think the processor is the problem. Try the answer [here](https://stackoverflow.com/a/34390517/7456923) to limit yourself to 1 thread (set `intra_op_parallelism_threads` *and `inter_op_parallelism_threads`* to `1`). It should be slower, but if that change alone makes you always get the same result, then you'll know that multi-threading is indeed the problem. – gdelab Jan 16 '18 at 16:07
  • Possible duplicate of [Tensorflow generate random seed](https://stackoverflow.com/questions/48235738/tensorflow-generate-random-seed) –  Jul 22 '19 at 22:29

0 Answers0