How to write a custom loss function in Tensorflow?

Question

I am new to tensorflow. I want to write my own custom loss function. Is there any tutorial about this? For example, the hinge loss or a sum_of_square_loss(though this is already in tf)? Can I do it directly in python or I have to write the cpp code?

Answer 1

We need to write down the loss function. For example, we can use basic mean square error as our loss function for predicted y and target y_:

 loss_mse = 1/n(Sum((y-y_)^2))

There are basic functions for tensors like tf.add(x,y), tf.sub(x,y), tf.square(x), tf.reduce_sum(x), etc.

Then we can define our loss function in Tensorflow like:

cost = tf.reduce_mean(tf.square(tf.sub(y,y_)))

Note: y and y_ are tensors.

Moreover, we can define any other loss functions if we can write down the equations. For some training operators (minimizers), the loss function should satisfy some conditions (smooth, differentiable ...).

In one word, Tensorflow define arrays, constants, variables into tensors, define calculations using tf functions, and use session to run though graph. We can define whatever we like and run it in the end.

Answer 2

In addition to the other answer, you can write a loss function in Python if it can be represented as a composition of existing functions.

Take a look, for example, at the implementation of sigmoid_cross_entropy_with_logits link, which is implemented using basic transformations.

Answer 3

Almost in all tensorflow tutorials they use custom functions. For example in the very beginning tutorial they write a custom function:

sums the squares of the deltas between the current model and the provided data

squared_deltas = tf.square(linear_model - y)
loss = tf.reduce_sum(squared_deltas)

In the next MNIST for beginners they use a cross-entropy:

cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))

---

As you see it is not that hard at all: you just need to encode your function in a tensor-format and use their basic functions.

For example here is how you can implement F-beta score (a general approach to F1 score). Its formula is:

The only thing we will need to do is to find how to calculate true_positive, false_positive, false_negative for boolean or 0/1 values. If you have vectors of 0/1 values, you can calculate each of the values as:

TP = tf.count_nonzero(actual, predicted)
FP = tf.count_nonzero((actual - 1) * predicted)
FN = tf.count_nonzero((predicted - 1) * actual)

Now once you know these values you can easily get your

denom = (1 + b**2) * TP + b**2 TN + FP
Fb = (1 + b**2) * TP / denom

Answer 4

def focal_loss(y_true, y_pred):
  pt_1 = tf.where(tf.equal(y_true, 1), y_pred, tf.ones_like(y_pred))
  pt_0 = tf.where(tf.equal(y_true, 0), y_pred, tf.zeros_like(y_pred))
  custom_loss=kb.square((pt_1-pt_0)/10)
  return custom_loss    

model.compile(loss=focal_loss,
          optimizer='adam',
          metrics=['accuracy'])

Answer 5

My realization of custom loss function with Kullbak-Leibler divergence where p = constant. I used it in autoencoder.

rho = 0.05

class loss_with_KLD(losses.Loss):
    
    def __init__(self, rho):
        super(loss_with_KLD, self).__init__()
        self.rho = rho
        self.kl = losses.KLDivergence()
        self.mse = losses.MeanSquaredError(reduction=tf.keras.losses.Reduction.SUM)
        
    def call(self, y_true, y_pred):
        mse = self.mse(y_true, y_pred)
        kl = self.kl(self.rho, y_pred)
        return mse + kl

May be helping to anyone.