4

The weights retrieved from restored model doesn't change and the input is also constant But the output of 'Relu:0' operation is giving different results each time.

Below is my code:

sess=tf.Session()   
saver = tf.train.import_meta_graph('checkpoints/checkpoints_otherapproach_1/cameranetwork_RAID_CNN-3100.meta')
saver.restore(sess,tf.train.latest_checkpoint(checkpoint_dir='checkpoints/checkpoints_otherapproach_1/'))

images = tf.get_default_graph().get_tensor_by_name('images:0')
phase = tf.get_default_graph().get_tensor_by_name('phase:0')
Activ = tf.get_default_graph().get_tensor_by_name('network/siamese_model/convolution_1/conv_1/Relu:0')

image_array = np.zeros(shape = [1,3,128,64,3]) #*******
imagepath = 'RAiD_Dataset' + '/images_afterremoving_persons_notinallcameras/'+'test'+'/camera_'+str(1)
fullfile_name = imagepath+"/"+ 'camera_1_person_23_index_1.jpg'
image_array[0][0] = cv2.imread(fullfile_name)
image_array[0][1] = image_array[0][0]
image_array[0][2] = image_array[0][0]
image_array = image_array.astype(np.float32)

feed_dict_values ={images: image_array, phase:False}
temp2 = sess.run(Activ, feed_dict =feed_dict_values)
temp1 = sess.run(Activ, feed_dict =feed_dict_values)

print (temp1==temp2).all()   #output is false
Trisoloriansunscreen
  • 1,543
  • 1
  • 15
  • 27
Gaurav Nayak
  • 41
  • 3

1 Answers1

4

There are two possible reasons for this:

  1. Some of the tensorflow ops inherit non-deterministic behavior from CUDA. This results in small numerical errors (which might be amplified by non-linearities). See this answer on how to try running your model on a single CPU thread. If the two arrays will turn out to be identical in this condition, then this is the case.

  2. I'm assuming that you know the graph you are loading, but the graph itself might produce inconsistent results 'by design' due to operations deliberately introducing either randomness or inconstant data. For example, consider operations that use the random number generator or operations that update variables (e.g., tf.assign) each time Activ is evaluated.

Community
  • 1
  • 1
Trisoloriansunscreen
  • 1,543
  • 1
  • 15
  • 27
  • 4
    One possible source of randomness might be dropout layers. – Peteris Apr 06 '18 at 10:53
  • The other possible source is activation function. In my case, I've trained on a graph which activation use mish function, but restore to a graph which use relu function. Because the graph structure is the same, you may not noticed that you have different activation function! – allenyllee May 06 '20 at 10:11