In TensorFlow is there any way to just initialize uninitialised variables?

Question

The standard way of initializing variables in TensorFlow is

init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)

After running some learning for a while I create a new set of variables but once I initialize them it resets all my existing variables. At the moment my way around this is to save all the variable I need and then reapply them after the tf.initalize_all_variables call. This works but is a bit ugly and clunky. I cannot find anything like this in the docs...

Does anyone know of any good way to just initialize the uninitialized variables?

Answer 1

There is no elegant* way to enumerate the uninitialized variables in a graph. However, if you have access to the new variable objects—let's call them v_6, v_7, and v_8—you can selectively initialize them using tf.initialize_variables():

init_new_vars_op = tf.initialize_variables([v_6, v_7, v_8])
sess.run(init_new_vars_op)

---

* A process of trial and error could be used to identify the uninitialized variables, as follows:

uninitialized_vars = []
for var in tf.all_variables():
    try:
        sess.run(var)
    except tf.errors.FailedPreconditionError:
        uninitialized_vars.append(var)

init_new_vars_op = tf.initialize_variables(uninitialized_vars)
# ...

...however, I would not condone such behavior :-).

Answer 2

UPDATE: TensorFlow 0.9 has a new method that "fixes" all this but only if you're using a VariableScope with reuse set to True. tf.report_uninitialized_variables which can be used in one line with sess.run( tf.initialize_variables( list( tf.get_variable(name) for name in sess.run( tf.report_uninitialized_variables( tf.all_variables( ) ) ) ) ) )

or more intelligently through the ability to specify the variables you expect to be initialized:

def guarantee_initialized_variables(session, list_of_variables = None):
    if list_of_variables is None:
        list_of_variables = tf.all_variables()
    uninitialized_variables = list(tf.get_variable(name) for name in
                                   session.run(tf.report_uninitialized_variables(list_of_variables)))
    session.run(tf.initialize_variables(uninitialized_variables))
    return unintialized_variables

This is still less ideal than actually knowing which variables are and are not initialized and taking care of that properly, but in the case of misdirection like the optim classes (see below) it may be hard to avoid.

Also note, tf.initialize_variables cannot evaluate tf.report_uninitialized_variables, so both of them have to be run within the context of the session to work.

---

There is an inelegant but concise way to do it. Before introducing your new variables run temp = set(tf.all_variables()) and afterwards run sess.run(tf.initialize_variables(set(tf.all_variables()) - temp)). These together will only initialize any variables created after the temp value is assigned.

I've been playing with transfer learning, so I wanted a quick way to do it too, but this is the best way I could find. Especially when using things like AdamOptimizer, which doesn't give you easy (or any, I'm not sure) access to the variables it uses. So the following actually shows up in my code. (I initialize the new layer's variables explicitly, and run it once to show the initial error before transfer learning. Just for a sanity check.)

temp = set(tf.all_variables())
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
#I honestly don't know how else to initialize ADAM in TensorFlow.
sess.run(tf.initialize_variables(set(tf.all_variables()) - temp))

And it solves all my problems.

EDIT: @Lifu_Huang's answer states the proper way to fix my problem. Theoretically, you should use tf.train.Optimizer.get_slot_names and tf.train.Optimizer.get_slot:

optim = tf.train.AdadeltaOptimizer(1e-4)
loss = cross_entropy(y,yhat)
train_step = optim.minimize(loss)
sess.run(tf.initialize_variables([optim.get_slot(loss, name)
                                  for name in optim.get_slot_names()])

This however gives me AttributeError: 'NoneType' object has no attribute 'initializer'. I'll make edits when I figure out what I did wrong, so you don't make my mistakes.

Answer 3

TF does not have a function that does exactly what you want, but you can easily write one:

import tensorflow as tf

def initialize_uninitialized(sess):
    global_vars          = tf.global_variables()
    is_initialized   = sess.run([tf.is_variable_initialized(var) for var in global_vars])
    not_initialized_vars = [v for (v, f) in zip(global_vars, is_initialized) if not f]
    
    print [str(i.name) for i in not_initialized_vars] # only for testing
    if len(not_initialized_vars):
        sess.run(tf.variables_initializer(not_initialized_vars))

Here I extract all global variables, iterate all of them and check whether they are already initialized. After this I get a list of uninitialized variables which I initialize. I also print variables that I am going to initialize for debugging purposes.

---

You can easily verify that it works as expected:

a = tf.Variable(3, name='my_var_a')
b = tf.Variable(4, name='my_var_b')

sess = tf.Session()
initialize_uninitialized(sess)
initialize_uninitialized(sess)

c = tf.Variable(5, name='my_var_a') # the same name, will be resolved to different name
d = tf.Variable(6, name='my_var_d')
initialize_uninitialized(sess)

print '\n\n', sess.run([a, b, c, d])

This will print all the unitialized variables before initializing them and the last sess.run will make sure persuade you that all variables are initialized.

---

You can also use tf.report_uninitialized_variables() to write a similar function. A sketch of it is here.

Answer 4

For the case @Poik mentioned, when variables are created by optimizers so that they cannot be accessed directly, a cleaner solution is to use tf.train.Optimizer.get_slot.

Some optimizer subclasses, such as MomentumOptimizer and AdagradOptimizer allocate and manage additional variables associated with the variables to train. These are called Slots. You can use tf.train.Optimizer.get_slot_names() to get all slot names a optimizer has and then use tf.train.Optimizer.get_slot to get the variable allocated for these slots.

Answer 5

I've come up with a method for TensorFlow r0.11:

def get_uninitialized_variables(variables=None):
    """Get uninitialized variables as a list.

    Parameters
    ----------
    variables : collections.Iterable[tf.Variable]
        Return only uninitialized variables within this collection.
        If not specified, will return all uninitialized variables.

    Returns
    -------
    list[tf.Variable]
    """
    sess = tf.get_default_session()
    if variables is None:
        variables = tf.all_variables()
    else:
        variables = list(variables)
    init_flag = sess.run(
        tf.pack([tf.is_variable_initialized(v) for v in variables]))
    return [v for v, f in zip(variables, init_flag) if not f]

Answer 6

Btw, if you want to initialize only a single tensor (for e.g. tf.Variable) that hasn't been initialized using tf.global_variables_initializer(), then you can use your_tensor.initializer in the sess.run() as in the following example:

In [196]: weights = tf.Variable(tf.zeros(shape=(3, 4)), name='weights')

In [197]: with tf.Session() as sess:
     ...:     sess.run(weights.initializer)
     ...:     print(weights.eval())
     ...:     

# the result
[[ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]]

Answer 7

I think that the easiest way is to create all the training operators first and initialize variables afterwards.

For example, I solved the problem of layer-wise pretraining with Adam Optimizer in the following way:

 # create an optimizer
 pretrain_optimizer =  tf.train.AdamOptimizer(learning_rate=learning_rate)

 # Make an array of the trainers for all the layers
 trainers=[pretrain_optimizer.minimize(loss_reconstruction(ae.run_less_layers(ae._input_, i+1), ae.run_less_layers(ae._input_, i+1, is_target=True)), global_step=tf.contrib.framework.get_or_create_global_step(), name='Layer_wise_optimizer_'+str(i)) for i in xrange(len(ae_shape) - 2)]

 # Initialize all the variables
 sess.run(tf.global_variables_initializer())