0

I have a model with multiple outputs, the losses for each output can have dependencies with one of the other outputs, as well as some masks computed from the data. The overall loss of the model is a weighted sum over the losses.

My model is subclassing tf.keras.Model and I am trying to write clean code that I can use with compile and fit. I would like the weights of the losses to be given during the compile.

One way I have found addressing the loss dependencies issue (after reading some documentation and this answer) is to feed the masks type of data as input of the model and, in the implementation of call, to add the loss of each output with Model.add_loss. Can someone confirm me this? How do I get y_true from there?

If this is a good solution, how do I specify that the overall model loss is a weighted sum of those losses during the compile, how can I access them?

Also would it be better to use add_loss on each layer in the implementation of the model's call? Same question, how do I access them during the compile?

If this was not a good solution, what is a good one?

import tensorflow as tf

class MyModel(tf.keras.Model):

    def __init__(self, base_trainable=False, feature_extractor=None, n=4, *kwargs):
        super(MyModel, self).__init__(*kwargs)
        if feature_extractor:
            self.feature_extractor = feature_extractor
        else:
            feature_extractor = tf.keras.applications.Resnet101(include_top=False,
                                                                weights='imagenet',
                                                                trainable=base_trainable)

        self.out1 = layers.Conv2D(n, kernel_size=(1,1), activation='sigmoid', name='out1')
        self.out2 = layers.Conv2D(n, kernel_size=(1,1), name='out2')
        self.out3 = layers.Conv2D(2*n, kernel_size=(1,1), 'out3')

    def call(self, inputs):
        img, mask1, mask2 = inputs
        x = self.feature_extractor(img)
        out1 = self.out1(x)
        out2 = self.out2(x)
        out3 = self.out3(x)
        # compute losses for each output? (but how do I access to each y_true?...)
        # ex:
        # 
        # model.add_loss(my_loss_for_out1(y1_true??
        #                                 out1,
        #                                 out2))
        # model.add_loss(my_loss_for_out1(y2_true??
        #                                 out2,
        #                                 mask1))
        # model.add_loss(my_loss_for_out1(y3_true??
        #                                 out3,
        #                                 mask2))


        return out1, out2, out3

model = MyModel()

model.compile(loss=???
              loss_weights=???)

Thank you

Nick Skywalker
  • 1,027
  • 2
  • 10
  • 26
  • You can read this example [link](https://www.pyimagesearch.com/2018/06/04/keras-multiple-outputs-and-multiple-losses/), I think, it is helpful, I used before. – Zaher88abd Jan 16 '20 at 15:24
  • Thank you, I had already looked at the link but the example use simple losses instead of custom losses that show dependencies on other outputs and mask data like in my example. – Nick Skywalker Jan 16 '20 at 15:36

1 Answers1

-1

You need to put a name for the output layers, then used the name to connect the output to the right loss function like this:

output_1 = SoftmaxLayer(X, name="output_1")
output_2 = SoftmaxLayer(X, name="output_2")    
model = Model(inputs=input, outputs=[output_1, output_2])    
losses = {
        "output_1": fun_1,
        "output_2": fun_2
    }

lossWeights = {"output_1": alpha, "output_2": beta}

model.compile(optimizer=opt, loss=losses, metrics=["acc"], loss_weights=lossWeights)
Zaher88abd
  • 448
  • 7
  • 20
  • Thanks for the answer. While this is easy when dealing with simple losses, in my case I don't know how to compute the losses outside of the model so that they can use other other outputs and other data (like a mask) than `y_true` and `y_pred`. I tried to illustrate in the example I gave. – Nick Skywalker Jan 16 '20 at 15:40
  • Each loss function is taking different 'y_true' and 'y_pred'. It is depends on the order in when you create the model 'Model(inputs=input, outputs=[output_1, output_2])'. – Zaher88abd Jan 16 '20 at 15:44
  • If you look at the commented line in my example, I try to make it clear that the losses have other dependencies than just `y_true` and `y_pred`. Let's take this one for example: `model.add_loss(my_loss_for_out1(y1_true??, out1, out2))`, here `my_loss_for_out1` depends on `y1_true`, `out1` (aka. `y1_pred`) but also `out2`. Some other ones also have dependencies that are not computed as output but are given by the data and fed into the network as dummy inputs. – Nick Skywalker Jan 16 '20 at 15:51
  • I didn't use 'model.add_loss' before but depends on this issue [link](https://github.com/tensorflow/tensorflow/issues/33471) there is but in this function with the last version TF2.0 – Zaher88abd Jan 16 '20 at 15:56
  • Sorry but this is not really answering my issue. – Nick Skywalker Jan 16 '20 at 15:58