10

I am loading from a saved model and I would like to be able to reset a tensorflow optimizer such as an Adam Optimizer. Ideally something like:

sess.run([tf.initialize_variables(Adamopt)])

or

sess.run([Adamopt.reset])

I have tried looking for an answer but have yet to find any way to do it. Here's what I've found which don't address the issue: https://github.com/tensorflow/tensorflow/issues/634

In TensorFlow is there any way to just initialize uninitialised variables?

Tensorflow: Using Adam optimizer

I basically just want a way to reset the "slot" variables in the Adam Optimizer.

Thanks

Community
  • 1
  • 1
Steven
  • 5,134
  • 2
  • 27
  • 38

4 Answers4

12

In tensorflow 2.x, e.g., Adam optimizer, you can reset it like this:

for var in optimizer.variables():
    var.assign(tf.zeros_like(var))
EdisonLeejt
  • 121
  • 1
  • 2
8

This question also bothered me for quite a while. Actually it's quite easy, you just define an operation to reset the current state of an optimizer which can be obtained by the variables() method, something like this:

optimizer = tf.train.AdamOptimizer(0.1, name='Optimizer')
reset_optimizer_op = tf.variables_initializer(optimizer.variables())

Whenever you need to reset the optimizer, run:

sess.run(reset_optimizer_op)

Official explanation of variables():

A list of variables which encode the current state of Optimizer. Includes slot variables and additional global variables created by the optimizer in the current default graph.

e.g. for AdamOptimizer basically you will get the first and second moment(with slot_name 'm' and 'v') of all trainable variables, as long as beta1_power and beta2_power.

Yuchi Yang
  • 81
  • 1
  • 2
  • 1
    I don't know if the following is still true. I heard that internally, the AdamOptimizer also track the number of steps it has run. And just doing `tf.variables_initializer(optimizer.variables())` is not enough. See LucasB's answer at https://stackoverflow.com/questions/41533489/how-to-initialise-only-optimizer-variables-in-tensorflow?noredirect=1&lq=1 – hamster on wheels Jan 22 '19 at 22:38
  • Same here. The variable_initializer is not enough – dtlam26 Aug 23 '21 at 09:03
4

The simplest way I found was to give the optimizer its own variable scope and then run 

optimizer_scope = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES,
                                 "scope/prefix/for/optimizer")
sess.run(tf.initialize_variables(optimizer_scope))

idea from freeze weights

Aaron
  • 2,354
  • 1
  • 17
  • 25
Steven
  • 5,134
  • 2
  • 27
  • 38
  • This does not appear to work with Adam in TensorFlow 1.0. – sunside Feb 19 '17 at 16:14
  • I tested it in tf 0.10 I'll rerun the test with tf 1.0. What was the error? – Steven Feb 20 '17 at 02:10
  • Should have specified - it doesn't list any trainable variables for Adam, the scope seems to be empty. – sunside Feb 20 '17 at 08:39
  • I'll take another look today. I just installed 1.0 myself then update the answer. there's also this which might help in the meantime http://stackoverflow.com/questions/35164529/in-tensorflow-is-there-any-way-to-just-initialize-uninitialised-variables – Steven Feb 20 '17 at 16:26
  • It should be tf.GraphKeys.GLOBAL_VARIABLES instead of tf. GraphKeys.TRAINABLE_VARIABLES. I'm going to edit the answer to fix it. – Aaron Jan 13 '18 at 18:56
0

Building upon @EdisonLeejt answer for Tensorflow 2.x, more generally you can first get the initial state (may not be zero e.g. if loaded from a checkpoint file) and then assign it i.e.

#Get initial states
init_states = [var.value() for var in optimizer.variables()]

#Do the optimization
...

#reset optimizer state to init_state
for val,var in zip(init_states,optimizer.variables()): var.assign(val)
tu_curious
  • 387
  • 4
  • 13