3

To perform early stopping in Tensorflow, tf.keras has a very convenient method which is a call tf.keras.callbacks, which in turn can be used in model.fit() to execute it. When we write Custom training loop, I couldn't understand how to make use of the tf.keras.callbacks to execute it. Can someone provide with a basic tutorial on how to do it?

https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch

https://machinelearningmastery.com/how-to-stop-training-deep-neural-networks-at-the-right-time-using-early-stopping/

vishal singh
  • 53
  • 5

1 Answers1

1

You have 2 approaches to create custom training loops.

One is this common 2 nested for loops.

or you can do this. All the callbacks and other features are available here

Tip : THE CODE BELLOW IS JUST AN SLICE OF CODE AND MODEL STRUCTURE IS NOT IMPLEMENTED. You should do it by your own.

More info? check here

class CustomModel(keras.Model):
  def train_step(self, data):
    # Unpack the data. Its structure depends on your model and
    # on what you pass to `fit()`.
    print(data)
    x, y = data
    with tf.GradientTape() as tape:
      y_pred = self(x, training=True)  # Forward pass
      # Compute the loss value
      # (the loss function is configured in `compile()`)
      loss = self.compiled_loss(y, y_pred,
                                regularization_losses=self.losses)
    # Compute gradients
    trainable_vars = self.trainable_variables
    gradients = tape.gradient(loss, trainable_vars)
    # Update weights
    self.optimizer.apply_gradients(zip(gradients, trainable_vars))
    # Update metrics (includes the metric that tracks the loss)
    self.compiled_metrics.update_state(y, y_pred)
    # Return a dict mapping metric names to current value
    return {m.name: m.result() for m in self.metrics}

# Construct and compile an instance of CustomModel
inputs = keras.Input(shape=(32,))
outputs = keras.layers.Dense(1)(inputs)
model = CustomModel(inputs, outputs)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['...'])

earlystopping_cb = keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True)

# Just use `fit` as usual
model.fit(train_ds, epochs=3, callbacks=[earlystopping_cb])

more info: https://keras.io/getting_started/intro_to_keras_for_engineers/#using-fit-with-a-custom-training-step

masoud parpanchi
  • 127
  • 2
  • 8
  • Can you complete a bit your example? Your code does not show what build_model is, and the call to CustomModel is commented. I am trying to implement this, but having some troubles calling CustomModel – SergioGM Feb 16 '21 at 09:32
  • 1
    @SergioGM I edited the answer and added a link for you. this one https://keras.io/guides/customizing_what_happens_in_fit/#wrapping-up-an-endtoend-gan-example You should implement model structure too. this is a subclassing approach for creating models – masoud parpanchi Feb 27 '21 at 07:19
  • 1
    The OP has specifically asked on using 'custom training loop' not the model.fit approach. Please correct your anwer. – Aakash Patil Apr 29 '21 at 12:27