1

I've got a partially trained model in Keras, and before training it any further I'd like to change the parameters for the dropout, l2 regularizer, gaussian noise etc. I have the model saved as a .h5 file, but when I load it, I don't know how to remove these regularizing layers or change their parameters. Any clue as to how I can do this?

akhetos
  • 686
  • 1
  • 10
  • 31
ѕняєє ѕιиgнι
  • 823
  • 3
  • 11
  • 32

3 Answers3

1

You can iterate over your pre trained keras model and remove kernel_regularizer(l1 or l2) from every layer where possible like this

def apply_regularization(
    model: tf.keras.Model,
    l1_regularization: Optional[bool],
    l2_regularization: Optional[bool],
) -> tf.keras.Model:
    for layer in model.layers:
        if hasattr(layer, "kernel_regularizer"):
            if l1_regularization:
                ## set to 0.0 to remove regularization
                layer.kernel_regularizer = tf.keras.regularizers.l1(0.0)
            if l2_regularization:
                layer.kernel_regularizer = tf.keras.regularizers.l2(0.0)
    return model
Saurabh Kumar
  • 2,088
  • 14
  • 17
0

Create a model with your required hyper-parameters and load the parameters to the model using load_weight().

Vigneswaran C
  • 461
  • 3
  • 14
  • But that doesn't save my optimizer state – ѕняєє ѕιиgнι Jun 21 '19 at 15:41
  • Optimizer state is equivalent to model parameters. For adaptive optimizers, you can start your second phase just by default values and optimizer senses it to adapt quickly. In case of SGD, even adaptation is not required and becomes quite direct. if you still need to use exact state you can follow https://stackoverflow.com/questions/49503748/save-and-load-model-optimizer-state – Vigneswaran C Jun 21 '19 at 15:50
0

Instead of saving the entire model to a .h5 file you could save the weights for each layer individually in your own format. e.g.

import pickle

# Create model and train ...

#save the weights for each layer in your model
network_config = {
    'layer1': layer1.get_weights(),
    'layer2': layer2.get_weights(),
    'layer3': layer3.get_weights()
}

with open('network_config.pickle', 'wb') as file:
    pickle.dump(network_config, file)

Then you can load only the weights for the layers you still use.

with open('network_config.pickle', 'rb') as file:
    network_config = pickle.load(file)

#build new model that may be missing some layers

layer1.set_weights(network_config['layer1'])
layer3.set_weights(network_config['layer3'])
rob
  • 17,995
  • 12
  • 69
  • 94