Issue with Tensorflow save and restore model

Question

I am trying to use the Transfer Learning approach. Here is a snapshot for the code where my code is learning over the Training data :

max_accuracy = 0.0
    saver = tf.train.Saver()
    for epoch in range(epocs):
        shuffledRange = np.random.permutation(n_train)
        y_one_hot_train = encode_one_hot(len(classes), Y_input)
        y_one_hot_validation = encode_one_hot(len(classes), Y_validation)
        shuffledX = X_input[shuffledRange,:]
        shuffledY = y_one_hot_train[shuffledRange]
        for Xi, Yi in iterate_mini_batches(shuffledX, shuffledY, mini_batch_size):
            sess.run(train_step,
                     feed_dict={bottleneck_tensor: Xi,
                                ground_truth_tensor: Yi})
            # Every so often, print out how well the graph is training.
            is_last_step = (i + 1 == FLAGS.how_many_training_steps)
            if (i % FLAGS.eval_step_interval) == 0 or is_last_step:
                train_accuracy, cross_entropy_value = sess.run(
                  [evaluation_step, cross_entropy],
                  feed_dict={bottleneck_tensor: Xi,
                             ground_truth_tensor: Yi})
                validation_accuracy = sess.run(
                  evaluation_step,
                  feed_dict={bottleneck_tensor: X_validation,
                             ground_truth_tensor: y_one_hot_validation})
                print('%s: Step %d: Train accuracy = %.1f%%, Cross entropy = %f, Validation accuracy = %.1f%%' %
                    (datetime.now(), i, train_accuracy * 100, cross_entropy_value, validation_accuracy * 100))
                result_tensor = sess.graph.get_tensor_by_name(ensure_name_has_port(FLAGS.final_tensor_name))
                probs = sess.run(result_tensor,feed_dict={'pool_3/_reshape:0': Xi[0].reshape(1,2048)})
                if validation_accuracy > max_accuracy :
                   saver.save(sess, 'models/superheroes_model')
                   max_accuracy = validation_accuracy
                   print(probs)
            i+=1

Here is where my code, where I am loading the model :

def load_model () :
    sess=tf.Session()    
    #First let's load meta graph and restore weights
    saver = tf.train.import_meta_graph('models/superheroes_model.meta')
    saver.restore(sess,tf.train.latest_checkpoint('models/'))
    sess.run(tf.global_variables_initializer())
    result_tensor = sess.graph.get_tensor_by_name(ensure_name_has_port(FLAGS.final_tensor_name))  
    X_feature = features[0].reshape(1,2048)        
    probs = sess.run(result_tensor,
                         feed_dict={'pool_3/_reshape:0': X_feature})
    print probs
    return sess

So now for the same data point I am getting totally different results while training and testing. Its not even close. During testing, my probabilities are near to 25% as I have 4 classes. But during training highest class probability is 90%.
Is there any issue while saving or restoring the model?

Issue resolved. I was training for large number of epoches, so probability decreased after some epoches. — neel, Jun 03 '17 at 14:16

score 2 · Answer 1 · answered Jun 03 '17 at 12:37

2

Be careful -- you are calling

sess.run(tf.global_variables_initializer())

after calling

saver.restore(sess,tf.train.latest_checkpoint('models/'))

I've done similar before, and I think that resets all your trained weights/biases/etc. in the restored model.

If you must, call the initializer prior to restoring the model, and if you need to initialize something specific from the restored model, do it individually.

answered Jun 03 '17 at 12:37

VS_FF

2,353
3
16
34

It did help but probability increases from 0.25 to 0.3 only. I checked for all the training points. – neel Jun 03 '17 at 13:50
I followed this to save and restore model https://stackoverflow.com/questions/33759623/tensorflow-how-to-save-restore-a-model – neel Jun 03 '17 at 13:56
Why don't you make a note of the key model values before you save it (weights/biases/etc.) and compare them with what you get after you restore? I've done this exercise before for some variables as a sanity check and for me it was OK, but who knows. – VS_FF Jun 03 '17 at 18:05
Another key issue to watch: are you sure you are restoring the right version of your model? It looks like you are saving it multiple times depending on some condition? Then I see some saved variables you refer to with ':0' You might want to investigate this... – VS_FF Jun 03 '17 at 18:08

score 2 · Answer 2 · answered Jun 03 '17 at 12:37

2

delete sess.run(tf.global_variables_initializer()) in your function load_model, if you do it, all your trained parameters will be replaced with the initial value that will produce 1/4 probability for each class

answered Jun 03 '17 at 12:37

Jie.Zhou

1,318
11
17

It did help but probability increases from 0.25 to 0.3 only. I checked for all the training points. – neel Jun 03 '17 at 13:50
I followed this to save and restore model https://stackoverflow.com/questions/33759623/tensorflow-how-to-save-restore-a-model – neel Jun 03 '17 at 13:56

Issue with Tensorflow save and restore model

2 Answers2