How to disable dropout AGAIN after setting training to true in Keras?

Question

I have the following code:

def create_keras_model(num_classes):
    """
    This function compiles and returns a Keras model.
    Should be passed to KerasClassifier in the Keras scikit-learn API.
    """
    input_shape = (28, 28, 1)
    
    
    x_in = keras.Input(shape=input_shape)
    x = layers.Conv2D(32, kernel_size=(3, 3), activation="relu")(x_in)
    x = layers.Dropout(0.25)(x,training=True)
    x = layers.MaxPool2D(pool_size=(2, 2))(x)
    x = layers.Conv2D(64, kernel_size=(3, 3), activation="relu")(x)
    x = layers.Dropout(0.25)(x,training=True)    
    x = layers.MaxPool2D(pool_size=(2, 2))(x)
    x = layers.Flatten()(x)
    x = layers.Dropout(0.5)(x,training=True)
    x = layers.Dense(num_classes)(x)

    model = Model(inputs=x_in, outputs=x)    

    model.compile(loss='categorical_crossentropy', 
                  optimizer='adam',
                  metrics=['accuracy'])

    return model

I need training=True for my purposes. However, after that purpose I need training=False in the Dropout-Layers. Is there a way to achieve that easily?

One way would be to load the model weights and load them into a second model that does not have any Dropout-Layer in the first place, but this seems over complicated.

Setting "trainable=False" like:

model.layers[-2].training = False
model.layers[-5].training = False
model.layers[-8].training = False

does not work. Calling predict several times on the same input data still yields different results.

AloneTogether · Accepted Answer · 2022-04-05T10:05:28.883

IIUC, you could try omitting the Dropout layers during inference by creating a new model:

import tensorflow as tf

def create_keras_model(num_classes):
    """
    This function compiles and returns a Keras model.
    Should be passed to KerasClassifier in the Keras scikit-learn API.
    """
    input_shape = (28, 28, 1)
    
    
    x_in = tf.keras.Input(shape=input_shape)
    x = tf.keras.layers.Conv2D(32, kernel_size=(3, 3), activation="relu")(x_in)
    x = tf.keras.layers.Dropout(0.25)(x,training=True)
    x = tf.keras.layers.MaxPool2D(pool_size=(2, 2))(x)
    x = tf.keras.layers.Conv2D(64, kernel_size=(3, 3), activation="relu")(x)
    x = tf.keras.layers.Dropout(0.25)(x,training=True)    
    x = tf.keras.layers.MaxPool2D(pool_size=(2, 2))(x)
    x = tf.keras.layers.Flatten()(x)
    x = tf.keras.layers.Dropout(0.5)(x,training=True)
    x = tf.keras.layers.Dense(num_classes)(x)

    model = tf.keras.Model(inputs=x_in, outputs=x)    

    model.compile(loss='binary_crossentropy', 
                  optimizer='adam',
                  metrics=['accuracy'])
    return model 

model = create_keras_model(1)

new_model = tf.keras.Sequential()

for idx, l in enumerate(model.layers):
  if not l.name.startswith('dropout'):
    if idx==0:
      new_model.add(tf.keras.layers.InputLayer(input_shape=(28, 28, 1)))
    new_model.add(l)

What you can also try doing is setting the dropout rate to zero during inference:

for layer in model.layers:
    if isinstance(layer, tf.keras.layers.Dropout) and hasattr(layer, 'rate'):
        layer.rate = 0.0

Also check the docs regarding the Dropout layer:

Note that the Dropout layer only applies when training is set to True such that no values are dropped during inference. When using model.fit, training will be appropriately set to True automatically, and in other contexts, you can set the kwarg explicitly to True when calling the layer.

(This is in contrast to setting trainable=False for a Dropout layer. trainable does not affect the layer's behavior, as Dropout does not have any variables/weights that can be frozen during training.)

As mentioned in the comments, you can create two models before the loop, one with and one without dropout and use this: model2.set_weights(model.get_weights()).

Sadly, this does not work. I tried this and then called predict on the model with the same input and the predicted results were different for each predict call. If it would work, they should be the same — Kev1n91, Apr 05 '22 at 09:33
model.predict(img) -> the img stays the same, call gets repeated like 20 times. All weights are static in inference and dropout should be too after setting training to false. Therefore the results should be exact the same everytime you call it when the input is exactly the same — Kev1n91, Apr 05 '22 at 09:36
Ok could be, as I said I did not really test it, why are you setting dropout to `True` manually? — AloneTogether, Apr 05 '22 at 09:41
To calculate uncertainty: https://stackoverflow.com/questions/43529931/how-to-calculate-prediction-uncertainty-using-keras This is a base requirement for my current purpose, it would make the question convoluted to explain — Kev1n91, Apr 05 '22 at 09:48
Hmm yes.. I see, then I guess creating a new model would be the 'simplest' option. See updated answer. — AloneTogether, Apr 05 '22 at 09:52
Sadly, I am doing this all together in a loop. This leads to heavy memory load on the GPU when creating the model anytime in the loop. So my current workaround is to create two models before the loop, one with one without dropout and use this: model2.set_weights(model.get_weights()). If you like you can add that to your answer — Kev1n91, Apr 05 '22 at 10:02

How to disable dropout AGAIN after setting training to true in Keras?

1 Answers1