2

How can I get the loss function used by tf.keras.Model.fit(x, y) to compare two outputs within the graph instead of one output with externally supplied target data, y?

Graph showing desired input to loss function

The manual says you can use tensors for target value which sounds like what I want but that you then also need the inputs to be tensors. But my inputs are numpy arrays and I don't think I should have to change that.

user1318499
  • 1,327
  • 11
  • 33
  • You could create another layer comparing the two tensors within the graph and then propagate the result (distance metric) to the output to be, e.g., minimised (target y = 0). – Max Oct 09 '19 at 10:41
  • But that output would then go through the "real" loss function, no? Seems possible but with a redundant loss calculation. It would be great if Model.fit() could just minimize the output instead of needing a loss function. – user1318499 Oct 09 '19 at 10:57
  • Exactly, solution 2 by Daniel Möller is what I meant. – Max Oct 09 '19 at 14:12
  • Just found out about option 3, take a look. It sounds pretty clean and correct. – Daniel Möller Oct 09 '19 at 17:57

1 Answers1

2

1 - Easy, best - maybe not good for memory

Why not just get the expected items for the loss already?

new_y_train = non_trainable_ops_model.predict(original_y_train)   
nn_model.fit(x_train, new_y_train)

This sounds definitely the best way if your memory can handle this. Simpler model, faster training.

You can even save/load the new data:

np.save(name, new_y_train)   
new_y_train = np.load(name)

2 - Make the model output the loss and use a dummy loss for compiling

Losses: 

def dummy_loss(true, pred):
    return pred

def true_loss(x):
    true, pred = x

    return loss_function(true, pred) #you can probably from keras.losses import loss_function

Model:

#given
nn_model = create_nn_model()
non_trainable_ops_model = create_nto_model()

nn_input = Input(nn_input_shape)
nto_input = Input(nto_input_shape)

nn_outputs = nn_model(nn_input)
nto_outputs = non_trainable_ops_model(nto_input)

loss = Lambda(true_loss)([nto_outputs, nn_outputs])

training_model = Model([nn_input, nto_input], loss)
training_model.compile(loss = dummy_loss, ...)

training_model.fit([nn_x_train, nto_x_train], np.zeros((len(nn_x_train),)))

3 - Use model.add_loss instead of compiling a loss

Following the same as the previous answer, you can:

training_model = Model([nn_input, nto_input], nn_outputs)

loss = true_loss([nto_outputs, nn_outputs])
training_model.add_loss(loss)

training_model.compile(loss=None, ...)
training_model.fit([nn_x_train, nto_x_train], None)

4 - Enable eager execution and make custom training loops

https://www.tensorflow.org/tutorials/customization/custom_training_walkthrough

Daniel Möller
  • 84,878
  • 18
  • 192
  • 214
  • I'm using option 3. It seems like a 5th way is to make a custom loss function wrapper to pass to Model.compile() according to this answer https://stackoverflow.com/a/56514531/1318499 . – user1318499 Oct 09 '19 at 22:52