3

I try to divide my neural network model and restore() function with setting random weights to zero. Here's the model code: http://pastebin.com/TqN6kkeb (it works properly).

And here's the function:

from __future__ import print_function

import tensorflow as tf
tf.GraphKeys.VARIABLES = tf.GraphKeys.GLOBAL_VARIABLES
import random
from LogReg import accuracy
from LogReg import W
from LogReg import x,y


# Import MNIST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)


def restore(model_file):

    with tf.Session() as sess:

        new_saver = tf.train.import_meta_graph(model_file + ".meta")
        new_saver.restore(sess, model_file)

        with tf.variable_scope("foo", reuse=True):
            temp_var = tf.get_variable("W")
            size_2a = tf.get_variable("b")

        size_2 = tf.shape(size_2a).eval()[0]
        size_1 = tf.shape(temp_var).eval()[0]

        ones_mask = tf.Variable(tf.ones([size_1, size_2]))
        arg = random.sample(xrange(size_1), size_1/2)
        index_num=tf.convert_to_tensor(arg, dtype=tf.int32)
        print("om", ones_mask)
        print("index", index_num)
        print(W)

        zeroes = tf.zeros([size_1/2, size_2])
        update = tf.scatter_update(ones_mask, index_num, zeroes)
        print(update)

        assign_op = W.assign(tf.mul(W, update))
        sess.run(update)
        sess.run(assign_op)
        init_op = tf.global_variables_initializer()
        sess.run(init_op)
        new_saver.save(sess, model_file)
        print("Accuracy_new:", accuracy.eval({x: mnist.test.images, y:mnist.test.labels}))

restore('./MyModel2')

The problems are: 1) is that it keeps writing me FailedPreconditionError (see above for traceback): Attempting to use uninitialized value Variable in this line: 

update = tf.scatter_update(ones_mask, index_num, zeroes)

no matter what. I have read these topics: Prettytensor: Attempting to use uninitialized value and Update a subset of weights in TensorFlow (and many others), but advices from there didn't help to fix my bug. And I don't understand, what's the problem with the initialization as long as I run tf.global_variables_initializer();

2) all of the weights seem to be setting to zero instead of the half, and I can't understand why.

Please, help, I really stuck.

Community
  • 1
  • 1
TheDoctor
  • 105
  • 1
  • 9
  • You have variables whose initializations depend on each other. This is a known issue. See [4920](https://github.com/tensorflow/tensorflow/issues/4920) for discussion and solutions – Yaroslav Bulatov Jan 31 '17 at 23:45
  • @YaroslavBulatov, thank you, that helped me to fix the first problem. However, I still don't quite understand why all the weights seem to be all zeroes. – TheDoctor Feb 01 '17 at 01:26
  • Not sure if this is related, but I've had an issue before where variables inside def: functions aren't affected by global_variables_initializer() and it helped to also call local_variable_initializer(). That seems to have helped... – VS_FF Feb 01 '17 at 10:55
  • Can you give the smallest possible, self-contained example that exhibits the remaining problem? The smaller it is, the more likely we will be able to help. – Peter Hawkins Feb 02 '17 at 21:14
  • @PeterHawkins, I want to set to zero random indices in all-ones matrix, and then multiply it on the weights matrix -> part of the weights are randomly set to zero, here is the small code snippet: http://pastebin.com/z1T87AQZ Thanks in advance. – TheDoctor Feb 02 '17 at 22:45
  • One further request: can you make the example self-contained? (i.e., something I can run, with just Tensorflow installed?) It's much easier to help with examples that you can actually run :-) – Peter Hawkins Feb 02 '17 at 22:50
  • @PeterHawkins, unfortunately, this whole function should be run( The point is, I need restore all weights and sizes from previously saved model, and to check the accuracy after the operations in the function are done, and it counts in LogReg file here:http://pastebin.com/TqN6kkeb Tensorflow is enough, though. – TheDoctor Feb 02 '17 at 23:27
  • @PeterHawkins, I have rewritten the code as simple as possible, but now I cannot monitor the accuracy, so I don't quite understand how to check how the code is working, print(update) doesn't show the internals of the variable:http://pastebin.com/yrPAzzbi – TheDoctor Feb 02 '17 at 23:47

1 Answers1

0

Just for the record (and others finding this post), the method name has changed, as per the page here: https://www.tensorflow.org/versions/r0.10/how_tos/variables/#initialization

you should run the initialize_all_variables() method like this:

import tensorflow as tf
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
Rishabh Agrahari
  • 3,447
  • 2
  • 21
  • 22
jesses.co.tt
  • 2,689
  • 1
  • 30
  • 49
  • initialize_all_variables() is deprecated, see https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/initialize_all_variables – S Wang Jun 07 '17 at 06:56