3

Problem: I want to monitor my model better during training. Because in some cases the loss suddenly turn to nan during training, and I want to know what the model is doing when this happens. Besides that, I want to see if a certain layer obeys a specific condition (rows and columns should sum to one).

Approach: Defining a custom metric wont help, since this one will only work on y_pred and y_true. Maybe there is some complex solution with building a model within a model, and trying to somehow calculate a metric on the output of the intermediate-model layer. But that feels a bit too complex.

Solution: The only thing I could think of otherwise is to switch to Tensorflow itself, so that I have more control over the training process. Any other ideas?

zwep
  • 1,207
  • 12
  • 26
  • 1
    You can just make a callback that does the computation you want and prints some values. – Dr. Snoopy Nov 05 '18 at 13:00
  • Thanks, that's it. – zwep Nov 05 '18 at 13:38
  • You could still wrap the custom metric within a function that has the intermediate tensor as argument. – rvinas Nov 05 '18 at 15:27
  • 1
    Really? Because this is what I get from the documentation "Custom metrics can be passed at the compilation step. The function would need to take (y_true, y_pred) as arguments and return a single tensor value." So I thought that I would be bounded by those input values – zwep Nov 05 '18 at 15:38

2 Answers2

2

The model.metrics_tensors.append does not work in TensorFlow 2.x

So if you're using the add_loss method, you can also use the model.add_metric method in Keras / TensorFlow 2.x.

For example, if we want to track the KL loss from the z_mean and z_log_var (output of an intermediate layer) in VAE we can do it this way:

kl_loss = 1 + z_log_var - K.square(z_mean) - K.exp(z_log_var) 
kl_loss = K.sum(kl_loss, axis=-1)
kl_loss *= -0.5

Then,

model.add_loss(vae_loss) 
model.add_metric(kl_loss,name='kl_loss')    
model.add_metric(reconstruction_loss,name='reconstruction_loss') 
model.compile(optimizer='adam')

Then,

Epoch 1/50
469/469 [==============================] - 3s 6ms/step - loss: 51.4340 - kl_loss: 4.5296 - reconstruction_loss: 46.9097 - val_loss: 42.0644 - val_kl_loss: 6.0029 - val_reconstruction_loss: 36.0615
Lina Achaji
  • 111
  • 1
  • 6
1

There are several ways to do this without the need to construct a callback, depending on how you add your losses.

If you add the loss with model.add_loss, you need to display it through a workaround by adding the metric after the compile step (as discussed here)

This results in something like this (specifically for a VAE, one is interested in kl_loss which depends on the intermediate layer): 

reconstruction_loss = mse(K.flatten(inputs), K.flatten(outputs))
kl_loss = beta*K.mean(- 0.5 * 1/latent_dim * K.sum(1 + z_log_var - K.square(z_mean) - K.exp(z_log_var), axis=-1))

model.add_loss(reconstruction_loss)
model.add_loss(kl_loss)
model.compile(optimizer='adam')

model.metrics_tensors.append(kl_loss)
model.metrics_names.append("kl_loss")

model.metrics_tensors.append(reconstruction_loss)
model.metrics_names.append("mse_loss")

For me this gives an output like this: 

Epoch 1/1
252/252 [==============================] - 23s 92ms/step - loss: 0.4336 - kl_loss: 0.0823 - mse_loss: 0.3513 - val_loss: 0.2624 - val_kl_loss: 0.0436 - val_mse_loss: 0.2188

If you don't use model.add_loss but pass your losses directly in the compiler, than you need to define a custom metric (similar to custom loss metric) and pass the metric to the compile step. In the case above: 

def customMetric(kl_loss):

    def klLoss(y_true,y_pred):    
        return kl_loss

    return klLoss

model.compile(..., metrics=[customMetric(kl_loss)])
dumkar
  • 735
  • 1
  • 5
  • 15