0

It seems setting model.trainable=False in tensorflow keras does nothing except for to print a wrong model.summary(). Here is the code to reproduce the issue:

import tensorflow as tf
import numpy as np
IMG_SHAPE = (160, 160, 3)

# Create the base model from the pre-trained model MobileNet V2
base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE,
                                               include_top=False, 
                                               weights='imagenet')
base_model.trainable = False
# for layer in base_model.layers:
#     layer.trainable=False
bc=[] #before compile
ac=[] #after compile
for layer in base_model.layers:
    bc.append(layer.trainable)
print(np.all(bc)) #True
print(base_model.summary()) ##this changes to show no trainable parameters but that  is wrong given the output to previous np.all(bc)
base_model.compile(optimizer=tf.keras.optimizers.Adam(lr=0.001), 
              loss='categorical_crossentropy', 
              metrics=['accuracy'])
for layer in base_model.layers:
    ac.append(layer.trainable)
print(np.all(ac)) #True
print(base_model.summary()) #this changes to show no trainable parameters but that  is wrong given the output to previous np.all(ac)

In light of this - What is the expected behavior and purpose of model.trainable=False in tensorflow keras?

figs_and_nuts
  • 4,870
  • 2
  • 31
  • 56

1 Answers1

0

https://github.com/tensorflow/tensorflow/issues/29535

I think this issue could help.

If you are looking for a way to not update some weights in your model I would suggest using the parameter var_list in the minimize function from your Optimizer.

For some reason when creating a model from keras Tensorflow switch all tf.Variables to True, and since all are Tensors we are not able to update the value to False.

What I do in my code is create scope names for all pretrained models and loop over it adding all layers that are not from my pretrained model. 

trainable_variables = []
variables_collection = tf.get_collection('learnable_variables')
for layer in tf.trainable_variables():
    if 'vgg_model' not in layer.name:
        trainable_variables.append(layer)
        tf.add_to_collection('learnable_variables', layer)

grad = tf.train.GradientDescentOptimizer(lr)
train_step = grad.minimize(tf.reduce_sum([loss]), var_list=trainable_variables)

Watch out for global_initializer as well, since it will overwrite your pretrained Weights as well. You can solve that by using tf.variables_initializer and passing a list of variables you want to add weights.

sess.run(tf.variables_initializer(variables_collection))

Source I used when trying to solve this problem Is it possible to make a trainable variable not trainable?

TensorFlow: Using tf.global_variables_initializer() after partially loading pre-trained weights

Nathan Gavenski
  • 123
  • 7