How can I implement this custom loss function in Keras?

Question

I'm trying to implement a custom loss function on my neural network, which would look like this, if tensors were, instead, numpy arrays:

def custom_loss(y_true, y_pred):
    activated = y_pred[y_true > 1]
    return np.abs(activated.mean() - activated.std()) / activated.std()

The y's have a shape of (batch_size, 1); that's to say, it's a scalar output for each input row.

obs: this post (Converting Tensor to np.array using K.eval() in Keras returns InvalidArgumentError) gave me an initial direction for which to walk on.

Edit:

This is a reproducible setup for which I'm trying to apply the custom loss function:

import numpy as np

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers


X = np.random.normal(0, 1, (256, 5))
Y = np.random.normal(0, 1, (256, 1))

model = keras.Sequential([
    layers.Dense(1),
])

model.compile(optimizer='adam', loss=custom_loss)

model.fit(X, Y)

The .fit() on the last line throws the error AttributeError: 'Tensor' object has no attribute 'mean', if I define custom_loss as stated above on my question.

Does this answer your question? [Make a custom loss function in keras](https://stackoverflow.com/questions/45961428/make-a-custom-loss-function-in-keras) — Innat, Feb 20 '21 at 18:17
@M.Innat, I'm not being able to adapt the answer you suggested to my case. I even tried to use the same code (the one from the answer) and I'm getting the error `TypeError: Input 'y' of 'Mul' Op has type bool that does not match type float32 of argument 'x'.` . Not sure if there is a version compatiblity problem between the newest tensorflow and the one there. — L. B., Feb 20 '21 at 18:42
Can you give some reproducible code? I faced such and if you can give some repro code, that would be nice. — Innat, Feb 20 '21 at 20:28

score 2 · Answer 1 · answered Feb 20 '21 at 03:40

2

Have you tried writing it in tensorflow as had gradient problems? Or is this just how to do so in tensorflow? -- Don't worry, I won't give you a classic toxic SO response! I would try something like this (not tested, but seems along the right track):

def custom_loss(y_true, y_pred):
    activated = tf.boolean_mask(y_pred, tf.where(y_true>1))
    return tf.math.abs(tf.reduce_mean(activated) - tf.math.reduce_std(activated)) / tf.math.reduce_std(activated))

You may need to play around with dimensions in there, since all of those functions allow for specifying the dimensions to work with.

Also, you will lose the loss function when you save the model, unless you subclass the general loss function. That may be more detail than you are looking for, but if you have problems saving and loading the model, let me know.

answered Feb 20 '21 at 03:40

alwaysmvp45

437
4
8

1

I tried using some of the tensorflow's tensor operators but I couldn't get it right. One of the errors that appeared, which is the same I'm getting on the first line of the function you suggested is this: `ValueError: Shapes (32, 1) and (None, 2) are incompatible`. Thanks for your friendliness. – L. B. Feb 20 '21 at 03:52
1

What are the shapes of y_true and y_pred? It looks like it is a broadcasting problem. The boolean_mask() function is very specific in the shapes between the tensor and the condition, so I would suspect that function is the one raising the errors. – alwaysmvp45 Feb 20 '21 at 05:12
When I call `print(y_true.shape, y_pred.shape)` on the top of the function, the outcome is `(32, 1), (32, 1)` – L. B. Feb 20 '21 at 15:53
1

is 32 the batch size? I am guessing it is having a problem determining batch size vs feature size if it is 2D. I would try either expanding the dimensions to (32, 1, 1) for both, so the batch size is obvious, or explicitly stating which dimensions you want to apply the boolean mask and following functions to – alwaysmvp45 Feb 20 '21 at 16:28
Yes, 32 is the batch_size (which comes by default). The features table is indeed 2D. I tried expanding the dimensions with `tf.expand_dims(y, axis=-1)`, so the shapes are now `(32, 1, 1)`. The error message changed slightly to `ValueError: Shapes (32, 1) and (None, 3) are incompatible` – L. B. Feb 20 '21 at 17:03

Innat · Accepted Answer · 2021-02-20T22:49:20.443

It's a simple catch. You can use your custom loss as follows

def custom_loss(y_true, y_pred):
    activated = y_pred[y_true > 1]
    return tf.math.abs(tf.reduce_mean(activated) - 
                       tf.math.reduce_std(activated)) / tf.math.reduce_std(activated)

or if you want to use tf.boolean_mask(tensor, mask, ..) then you need to ensure that the mask condition is in the shape of (None,) or 1D. And if we apply tf.where(y_true>1) it will produce a 2D tensor that needs to be reshaped in your case.

def custom_loss(y_true, y_pred):
    activated = tf.boolean_mask(y_pred, tf.reshape(tf.where(y_true>1),[-1]) )
    return tf.math.abs(tf.reduce_mean(activated) - 
                       tf.math.reduce_std(activated)) / tf.math.reduce_std(activated)

How can I implement this custom loss function in Keras?

2 Answers2