3

I built a model that uses a pre-trained VGG-16 as a base network and then adds a few layers on top of that.

At training time, my model consists partially of variables that belong to the pre-trained VGG-16, i.e. variables that are already initialized (the entire model is being loaded from a SavedModel protocol buffer), and partially of uninitialized variables from the layers I added.

Before I start the training, I have to initialize the variables of the layers I added on top of the pre-trained VGG-16. I do that by running tf.global_variables_initializer().

Here is the question:

If tf.global_variables_initializer() initializes all global variables, why does it not override the pre-trained weights with their initialization values?

Alex
  • 3,316
  • 4
  • 26
  • 52

2 Answers2

1

In fact, it does override pre-trained weights:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import backend

session = tf.Session ()
backend.set_session (session)
with session.as_default ():
    net = keras.applications.InceptionV3 (input_shape = (3, 299, 299))
    print ('weights before: ', net.get_layer ('conv2d_1').get_weights ())

    session.run (tf.global_variables_initializer ())

    print ('weights after: ', net.get_layer ('conv2d_1').get_weights ())

Output:

weights before:  [array([[[[ 7.42468759e-02,  7.99097791e-02, -1.20985091e-01, ...,
      -2.42504068e-02, -8.17770883e-03, -7.11583123e-02],
     [ 5.46021610e-02,  1.13289319e-01,  3.99106629e-02, ...,
       2.37093717e-02, -6.01868033e-02, -1.08440826e-02],
     [ 1.00897752e-01,  1.96732566e-01,  8.93706456e-02, ...,
       1.19806841e-01, -9.84039158e-02, -7.20797330e-02],

weights after:  [array([[[[-7.92664364e-02, -5.87778017e-02, -9.76844281e-02, ...,
       9.62431133e-02,  7.16331154e-02, -3.22120935e-02],
     [-6.17715716e-03, -6.95393384e-02,  9.68316942e-02, ...,
       5.01702428e-02,  8.89533758e-02,  9.80506092e-02],
     [-1.89049393e-02, -4.31398787e-02, -9.45695043e-02, ...,
       7.52686858e-02,  7.27956891e-02, -3.26380134e-03],

To overcome this, put custom variables in a separate collection:

my_variables_collection = tf.get_collection ('my_variables')

t = tf.Variable (0.0, name='adam_t', trainable=False)
tf.add_to_collection ('my_variables', t)

m = tf.Variable (backend.zeros ((3, 299, 299)), name='adam_m', trainable=False)
tf.add_to_collection ('my_variables', m)

# only initialize variables from my list:
session.run (tf.variables_initializer (my_variables_collection))
jaros
  • 46
  • 3
1

Saving and loading weights before and after variable initializer worked for me

weights=model.get_weights
sess=tf.Session()
K.set_session(sess)
sess.run(tf.global_variables_initializer ())
model.set_weights(weights)
print(np.argmax(sess.run(model(preprocessed_input))))
user760664
  • 171
  • 2
  • 5