16

I am writing a keras custom loss function where in I want to pass to this function the following: y_true, y_pred (these two will be passed automatically anyway), weights of a layer inside the model, and a constant.

Something like below:

def Custom_loss(y_true, y_pred, layer_weights, val = 0.01):
    loss = mse(y_true, y_pred)
    loss += K.sum(val, K.abs(K.sum(K.square(layer_weights), axis=1)))
    return loss

But the above implementation gives me error. How can I achieve this in keras ?

Marco Cerliani
  • 21,233
  • 3
  • 49
  • 54
Uday Shankar S
  • 317
  • 1
  • 3
  • 10

2 Answers2

25

New answer

I think you're looking exactly for L2 regularization. Just create a regularizer and add it in the layers:

from keras.regularizers import l2

#in the target layers, Dense, Conv2D, etc.:
layer = Dense(units, ..., kernel_regularizer = l2(some_coefficient))

You can use bias_regularizer as well.
The some_coefficient var is multiplied by the square value of the weight.

PS: if val in your code is constant, it should not harm your loss. But you can still use the old answer below for val.

Old answer

Wrap the Keras expected function (with two parameters) into an outer function with your needs:

def customLoss(layer_weights, val = 0.01):
    
    def lossFunction(y_true,y_pred):    
        loss = mse(y_true, y_pred)
        loss += K.sum(val, K.abs(K.sum(K.square(layer_weights), axis=1)))
        return loss

    return lossFunction

model.compile(loss=customLoss(weights,0.03), optimizer =..., metrics = ...)

Notice that layer_weights must come directly from the layer as a "tensor", so you can't use get_weights(), you must go with someLayer.kernel and someLayer.bias. (Or the respective var name in case of layers that use different names for their trainable parameters).

The answer here shows how to deal with that if your external vars are variable with batches: How to define custom cost function that depends on input when using ImageDataGenerator in Keras?

Community
  • 1
  • 1
Daniel Möller
  • 84,878
  • 18
  • 192
  • 214
  • Is it a coincidence, that customLoss has also exactly two input variables? Or is the wrapped function again limited to two input parameters? – AlexConfused Aug 22 '18 at 16:30
  • 1
    The "lossFunction" must always have 2 params, ground truth and predictions. The wrapper (outer) function is irrelevant. – Daniel Möller Aug 22 '18 at 16:38
  • It doesn't seem to work with model loading after the model is saved. It requires the parameters to be passed again, which cannot be done for tensors at least – HitLuca Jan 11 '19 at 13:53
  • 1
    Yes... saving and loading in keras is very complicated if you start to get too far from the standard models. It needs hacky workarouds. Sometimes I prefer to rebuild the entire model (that means I keep the model's code) and save/load only the weights. – Daniel Möller Jan 12 '19 at 19:09
  • @DanielMöller Yes and that's ridiculous, why is saving custom models not a keras priority? It is indeed often impossible to save and load custom models and this worked fine in TF1 where we didn't rely on keras models for saving, just had to build the graph and save. Now in TF2 we are forced to try to make it suit the keras model standards and we have to handle every single custom function or class manually to try and get it saved. As if tensorflow abandoned the usability of models in production – Kristof Sep 01 '19 at 20:21
  • The very reason why I bothered to refactor my code into keras functional API is because I hoped once I get it 'regularized' this will allow me to save my models easily. Now hello, like usually, in the end there's a catch, you just can't add custom loss functions and save the whole model. Back to square 1, I shouldn't have bothered then. Currently there's no way in TF2 to save and load entire VAE models for production – Kristof Sep 01 '19 at 20:25
  • 1
    But you can. You just need to pass the loss function to `custom_objects` when you are loading the model. But remember to pass "everything" that keras may not know, from weights to the loss itself. You must keep your custom loss code. By the way, if the idea is to "use" the model, you don't need loss, optimizer, etc.. These are only for training. – Daniel Möller Sep 01 '19 at 20:34
  • @DanielMöller The above solution may not work already in TF2. I'm getting: TypeError: An op outside of the function building code is being passed a "Graph" tensor. It is possible to have Graph tensors leak out of the function building context by including a tf.init_scope in your function building code. – Kristof Sep 01 '19 at 21:22
  • Sorry from the error above it's likely my issue is not related to your solution itself but has to do with the fact that one of the variables used in my custom loss function is the output of a custom keras layer that includes a random sampling step (tf.keras.backend.random_normal). I think that's what it refers to as 'An op outside of the function building code is being passed a "Graph" tensor.' ... Not sure – Kristof Sep 01 '19 at 21:34
  • I'm not sure... sounds like that's it. My guess is that some variable is being initialized outside the model, I'm not sure how to solve it... – Daniel Möller Sep 01 '19 at 21:38
  • @Kristor did you solve the TypeError problem you describe? I try to implement a vae and also think the error refers to the random sample mean and log var being inputs to the loss function. – mchl_k Jan 27 '20 at 19:44
1

You can do this another way by using the lambda operator as following:

model.compile(loss= [lambda y_true,y_pred: Custom_loss(y_true, y_pred, val=0.01)], optimizer =...)

There are some issues regarding saving and loading the model this way. A workaround is to save only the weights and use model.load_weights(...)

Akylas Stratigakos
  • 11
  • 1