3

I'm reading an article about tuning hyperparameters in keras tuner. It includes code to build a model that has this code:

def build_model(hp):
    """
    Builds model and sets up hyperparameter space to search.
    
    Parameters
    ----------
    hp : HyperParameter object
        Configures hyperparameters to tune.
        
    Returns
    -------
    model : keras model
        Compiled model with hyperparameters to tune.
    """
    # Initialize sequential API and start building model.
    model = keras.Sequential()
    model.add(keras.layers.Flatten(input_shape=(28,28)))
    
    # Tune the number of hidden layers and units in each.
    # Number of hidden layers: 1 - 5
    # Number of Units: 32 - 512 with stepsize of 32
    for i in range(1, hp.Int("num_layers", 2, 6)):
        model.add(
            keras.layers.Dense(
                units=hp.Int("units_" + str(i), min_value=32, max_value=512, step=32),
                activation="relu")
            )
        
        # Tune dropout layer with values from 0 - 0.3 with stepsize of 0.1.
        model.add(keras.layers.Dropout(hp.Float("dropout_" + str(i), 0, 0.3, step=0.1)))
    
    # Add output layer.
    model.add(keras.layers.Dense(units=10, activation="softmax"))
    
    # Tune learning rate for Adam optimizer with values from 0.01, 0.001, or 0.0001
    hp_learning_rate = hp.Choice("learning_rate", values=[1e-2, 1e-3, 1e-4])
    
    # Define optimizer, loss, and metrics
    model.compile(optimizer=keras.optimizers.Adam(learning_rate=hp_learning_rate),
                  loss=keras.losses.SparseCategoricalCrossentropy(),
                  metrics=["accuracy"])
    
    return model

I'm confused about what the numbers, 1,2 and 6 mean in the range function for the num_layers line.

Yoseph Ismail
  • 109
  • 7

1 Answers1

1

If you see the docs, the 2 and 6 are referring to the min and max values respectively. Also note:

[...] max_value is included in the possible values this parameter can take on

So this line:

for i in range(1, hp.Int("num_layers", 2, 6)):

basically means: generate a x number of Dense layers, where x is between 1 and 5 and the range from 2 to 6 is the hyperparameter you are working on / tuning.

AloneTogether
  • 25,814
  • 5
  • 20
  • 39
  • do you mean ```x``` is between 1 and 5 and therefore the range is from 2 to 6? so for example if i wanted the number of hidden layers to be between 5 and ten the line would be: ```for i in range(5, hp.Int("num_layers", 6, 11)):``` – Yoseph Ismail Jul 28 '22 at 10:52
  • okay I understand now, thankyou. So if I want at least 5 layers and at most 10 layers, all of which are tunable it is ```for i in range(5, hp.Int("num_layers", 5, 10)):``` – Yoseph Ismail Jul 28 '22 at 11:13
  • @YosephIsmail No problem. See my other comment and updated answer. – AloneTogether Jul 28 '22 at 11:14