Make a Custom loss function in Keras in detail

Question

I try to make a custom loss function in Keras.

I want to make this loss function

The dimension of output is 80. Batch size is 5000.

So I build this loss function below. But this doesn't work.

def normalize_activation(y_true, y_pred):
    nb_divide = K.reshape(K.sqrt(K.sum(K.square(y_pred), axis=1)),(5000, 1))
    nb_divide=numpy.tile(nb_divide,80)
    predicted=numpy.divide(y_pred,nb_divide)
    return K.sum(K.square(y_true-predicted))

ValueError: setting an array element with a sequence.

This error occurs. I think that the shape of y_true, y_pred is (5000,80).

where should I fix it??

It would be equivalent to use `losses.mean_squared_error(y_true, K.l2_normalize(y_pred, axis=-1))`. — Yu-Yang, Sep 20 '17 at 12:57
Or you can add a `Lambda(lambda x: K.l2_normalize(x, axis=-1))` layer at the end of your model, and use `loss='mse'`. — Yu-Yang, Sep 20 '17 at 13:14
@Yu-Yang Thanks a lot! When use lambda, weight in last link should be 1. I can't make it. so I'm using your first solution. thank you. but I wonder meaning of axis=-1. I know axis=0, axis=1. — Lucas Kim, Sep 22 '17 at 11:32
`axis=-1` is just “the last axis”, in this case it’ll be equivalent to `axis=1` so you can use either one. — Yu-Yang, Sep 22 '17 at 11:47

score 5 · Accepted Answer · answered Sep 20 '17 at 12:19

Loss functions should avoid all kinds of operations that are not from keras backend. The values are tensors and you must keep them like tensors.

And you don't need to reshape things unless you actually want them to behave in a specific way.

If you have shapes (5000,80) and (5000,1), you can make operations with them without needing K.repeat_elements() (the equivalent to numpy.tile).

So, supposing that 5000 is the batch size (number of samples) and 80 is the only actual dimension belonging to a sample:

def normalize_loss(yTrue,yPred):
    nb_divide = K.sqrt(K.sum(K.square(yPred),axis=1,keepdims=True)) 
        #keepdims=True keeps the shape like (5000,1)
        #this is not summing the entire batch, but only a single sample, is that correct?

    predicted = yPred/nb_divide
    return K.sum(K.square(yTrue-predicted))

Some observations:

(I'm not a loss function expert here) You're dividing only the predicted part, but not the true part. Wouldn't that create big differences between both values and result in a misguiding loss function? (Again, I'm not the expert here)
Usually people use K.mean() at the end of the loss function, but I see you used K.sum(). This is not a problem and doesn't prevent training from working. But you may like to visualize this same loss function for data with different sizes, and be able to compare them size-independently.

Make a Custom loss function in Keras in detail

1 Answers1

Linked