8

Basically, I want my custom loss function to alternate between usual MSE and a custom MSE that subtract values from different indexes.

To clarify, let's say I have a y_pred tensor that is [1, 2, 4, 5] and a y_true tensor that is [2, 5, 1, 3]. In usual MSE, we should get:

return K.mean(K.squared(y_pred - y_true))

That would do the following:

[1, 2, 4, 5] - [2, 5, 1, 3] = [-1, -3, 3, 2]

[-1, -3, 3, 2]² = [1, 9, 9, 4]

mean([1, 9, 9, 4]) = 5.75

I need my custom loss function to select the minimum value between this mean and other that switches indexes 1 and 3 from the y_pred tensor, i.e.:

[1, 5, 4, 2] - [2, 5, 1, 3] = [-1, 0, 3, 1]

[-1, 0, 3, 1]² = [1, 0, 9, 1]

mean([1, 0, 9, 1]) = 2.75

So, my custom loss would return 2.75, which is the minimum value between both means. To do this, I tried to transform y_true and y_pred tensors in numpy arrays, doing all math related as following:

def new_mse(y_true, y_pred):
    sess = tf.Session()

    with sess.as_default():
        np_y_true = y_true.eval()
        np_y_pred = y_pred.eval()

        np_err_mse = np.empty(np_y_true.shape)
        np_err_mse = np.square(np_y_pred - np_y_true)

        np_err_new_mse = np.empty(np_y_true.shape)
        l0 = np.square(np_y_pred[:, 2] - np_y_true[:, 0])   
        l1 = np.square(np_y_pred[:, 3] - np_y_true[:, 1])
        l2 = np.square(np_y_pred[:, 0] - np_y_true[:, 2])
        l3 = np.square(np_y_pred[:, 1] - np_y_true[:, 3])   
        l4 = np.square(np_y_pred[:, 4] - np_y_true[:, 4])
        l5 = np.square(np_y_pred[:, 5] - np_y_true[:, 5])
        np_err_new_mse = np.transpose(np.vstack(l0, l1, l2, l3, l4, l5))

        np_err_mse = np.mean(np_err_mse)
        np_err_new_mse = np.mean(np_err_new_mse)

        return np.amin([np_err_mse, np_err_new_mse])

Problem is that I can't use eval() method with y_true and y_pred tensors, not sure why. Finally, my questions are:

  1. Is there an easier method to work with indexing inside tensors and loss functions? I'm newbie to Tensorflow and Keras in general and I strongly believe transforming everything in numpy arrays isn't the optimal method at all.
  2. Not totally related to question, but when I tried to print y_true tensor's shape with K.shape(y_true), I got "Tensor("Shape_1:0", shape=(2,), dtype=int32)". That confuses me, since I'm working with a y.shape equal to (7032, 6), that is 7032 images with 6 labels each. Probably may be some misinterpretation related to my y and y_pred used by loss function.
Mateus Hora
  • 117
  • 1
  • 6

2 Answers2

3

Often you work just with backend functions, and you never try to know the actual values of the tensors. 

from keras.losses import mean_square_error

def new_mse(y_true,y_pred): 

    #swapping elements 1 and 3 - concatenate slices of the original tensor
    swapped = K.concatenate([y_pred[:1],y_pred[3:],y_pred[2:3],y_pred[1:2]])
    #actually, if the tensors are shaped like (batchSize,4), use this:
    #swapped = K.concatenate([y_pred[:,:1],y_pred[:,3:],y_pred[:,2:3],Y_pred[:,1:2])

    #losses
    regularLoss = mean_squared_error(y_true,y_pred)
    swappedLoss = mean_squared_error(y_true,swapped)

    #concat them for taking a min value
    concat = K.concatenate([regularLoss,swappedLoss])

    #take the minimum
    return K.min(concat)

So, for your items:

  1. You're totally right. Avoid numpy at all costs in tensor operations (loss functions, activations, custom layers, etc.)

  2. A K.shape() is also a tensor. It probably has shape (2,) because it has two values, one value will be 7032, the other value will be 6. But you can only see these values when you eval this tensor. Doing this inside loss functions is often a bad idea.

Daniel Möller
  • 84,878
  • 18
  • 192
  • 214
  • So, I'm getting "TypeError: Expected int32, got list containing Tensors of type '_Message' instead" in K.concatenate() method. I tried to change the order of arguments due to 'axis' parameter, but it doesn't seem to be the problem. Currently using Keras 1.2.2 and Tensorflow 1.1.0. Any ideas on how to solve this? – Mateus Hora Sep 15 '17 at 13:33
  • Hmmmm... I've read around that there are some significant differences between keras 1 and keras 2. I'm using keras 2.0.x. --- Please notice the little comment I added in the `swapped` line of my answer. You'll probably need to use the commented line. – Daniel Möller Sep 15 '17 at 13:35
  • You can also try to replace the two `regularLoss` and `swappedLoss` lines by `K.mean(K.square(y_pred - y_true))` and `K.mean(K.square(swapped - y_true))`, just in case these `message` tensors are coming from the imported loss functions. – Daniel Möller Sep 15 '17 at 13:42
  • I think I've fixed it by calling the `K.concatenate()` method directly from Tensorflow, that is `tf.concat()`, I understand that the code might not work using Theano, but it's ok for testing purposes. But I have a final question, the loss function is supposed to return an integer, like `0.082`, in example, or a tensor? I ask that because `K.min()` method returns a tensor and I think this could cause some problems in coding. – Mateus Hora Sep 15 '17 at 16:51
  • 1
    It can be a tensor, but `K.min()` will be a tensor with only one element. – Daniel Möller Sep 15 '17 at 16:56
  • 1
    (But I do suggest you move to keras 2, though) – Daniel Möller Sep 15 '17 at 16:56
2

If using Keras 2 you should just use the K.gather function to perform indexing.

Daniel Möller's answer becomes:

from keras.losses import mean_square_error

def reindex(t, perm):
    K.concatenate([K.gather(t, i) for i in perm])

def my_loss(y_true,y_pred):

    #losses
    regularLoss = mean_squared_error(y_true, y_pred)
    swappedLoss = mean_squared_error(y_true, reindex(y_pred, [0,3,2,1]))

    #concat them for taking a min value
    concat = K.concatenate([regularLoss,swappedLoss])

    #take the minimum
    return K.min(concat)
hawk78
  • 41
  • 3