Tensorflow same code but get different result from CPU device to GPU device

Question

I am trying to implement a program to test the Tensorflow performance on GPU device. Data test is MNIST data, supervised training using Multilayer perceptron(Neural networks). I followed this simple example but I change the number of performance batch gradient to 10000

for i in range(10000) :
batch_xs, batch_ys = mnist.train.next_batch(100)
sess.run(train_step,feed_dict={x : batch_xs, y_ : batch_ys})
if i % 500 == 0:
    print(i)

Eventually, when I check the predict accuracy using this code

correct_prediction = tf.equal(tf.argmax(y,1),tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction,"float"))
print(sess.run(accuracy,feed_dict={x:mnist.test.images,y_:mnist.test.labels}))
print(tf.convert_to_tensor(mnist.test.images).get_shape())

it turns out that the accuracy rate is different from CPU to GPU: when GPU returns the accuracy rate approximately 0.9xx while CPU returns only 0.3xx. Does anyone know the reason? or why can that issue happen?

TensorBoard would be useful for debugging this issue. – Yao Zhang Apr 05 '17 at 07:12 — Yao Zhang, Apr 05 '17 at 07:12

score 8 · Answer 1 · answered Apr 06 '17 at 19:41

There are two primary reasons for this kind of behavior (besides bugs).

Numerical stability

It turns out that adding numbers is not entirely as easy as it might seem. Let's say I want to add a trillion 2's together. The correct answer is two trillion. But if you add these together in floating point on a machine with a wordsize of only, say 32 bits, after a while, your answer will get stuck at a smaller value. The reason is that after a while, the 2's that you're adding are below the smallest bit of the mantissa of the floating point sum.

These kinds of issues abound in numerical computing, and this particular discrepancy is known in TensorFlow (1,2, to name a few). It's possible that you're seeing an effect of this.

Initial conditions

Training a neural nets is a stochastic process, and as such, it depends on your initial conditions. Sometimes, especially if your hyperparameters are not tuned very well, your net will get stuck near a poor local minima, and you'll end up with mediocre behavior. Adjusting your optimizer parameters (or better, using an adaptive method like Adam) might help out here.

Of course, with all that said, this is a fairly large difference, so I'd double check your results before blaming it on the underlying math package or bad luck.

I agree with your first statement, Very useful. I assume that that problem typically doesn't happen often. However it turns out my problem is initialization incorrect — Tran Minh, Apr 13 '17 at 02:59
In order to have reproducible results, one can put ```np.random.seed(1) from tensorflow import set_random_seed set_random_seed(2)``` in the code. — bio, Dec 18 '17 at 12:49

Tensorflow same code but get different result from CPU device to GPU device

1 Answers1

Linked