1

I try to solve a semantic segmentation problem with Keras and Tensorflow2 backend. I try to label each pixel as one of 22 class with categorical cross entropy. Shapes of my input and outputs are

Input: (None, 224, 224, 3)

Output: (None, 224, 224, 23) 22 and 1 for background

I wanted to add weights for each sample to try pseudo labeling with my model. For sample weights, I tried to create a sample weight array which is a 1D array and has the length same as batch size. But it failed and gave the below error:

weights can not be broadcast to values. values.rank=3. weights.rank=1.

Then I tried to give 3D array (16, 224, 224) as sample weights with the batch size of 16 and it gave the below error:

Found a sample_weight array with shape (16, 224, 224).
In order to use timestep-wise sample weights, you should specify sample_weight_mode="temporal" in compile(). 
If you just mean to use sample-wise weights, make sure your sample_weight array is 1D.
Burak Bekci
  • 23
  • 6
  • I'm having a similar problem trying to weight different images in tf.keras. Have you made any progress? – dpoiesz Aug 28 '20 at 15:49
  • For my case, no I could not make any progress. But If your outputs are just scalar class labels rather than 2d arrays, you can use sample_weight parameter of the fit method. If you have a custom data generator then return your sample weight array in the __get_item__ method of the datagen for each batch. – Burak Bekci Aug 30 '20 at 11:36

1 Answers1

0

I managed to solve a similar problem with TF2 with multiple outputs and using loss_weights in model.compile. In this way, the final loss used to train the network is a weighted sum of each loss. In my case I had 2 classes rather than 23, so you would need to modify the code for those 23. Note also that layer c9 in the code has 3 filters so you may need to increase it.

from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input
from tensorflow.keras.layers import Conv2D
from tensorflow.keras.losses import BinaryCrossentropy
from tensorflow.keras.optimizers import Adam
# build your CNN
inputs = Input((224, 224, 1), name='inputs')
c1 = Conv2D(64, (3, 3), activation='relu', padding='same')(inputs)
...
c9 = Conv2D(3, (3, 3), activation='relu',padding='same')(c9)

output_0 = Conv2D(1, (1, 1), name='class0')(c9)
output_1 = Conv2D(1, (1, 1), name='class1')(c9)

model = Model(inputs=inputs, outputs=[output_0, output_1])
model.compile(optimizer=Adam(learning_rate=0.001, name='adam'),
              loss=[BinaryCrossentropy(from_logits=True), BinaryCrossentropy(from_logits=True)],
              loss_weights=[1, 1000],
              metrics=[["accuracy"], ["accuracy", "mse"]])

Also, you could define a sampling pdf in the data generator so you feed the network with more objects belonging to one class than to the background, for example. The effect could be similar to the use of a weighted loss function. At the end, the network tries to minimize an averaged loss value, so if you add more samples of one class, this averaged loss value will be affected precisely by the result for that class.

eLearner
  • 61
  • 5
  • I didn't understand your approach, what is the ground truth values of the second output? – Burak Bekci Sep 19 '20 at 11:15
  • For a binary segmentation, the first output is the prediction of the foreground, and the second one is the prediction of the background (1-foreground). Instead of having an output with 2 channels and using the categorical cross-entropy, you have two outputs with one channel each and you use the binary cross-entropy (CCE) for each. If you use loss_weights = [1,1] you would have similar results as with the CCE. However, if you use loss_weights = [10, 1], then an error in the prediction of foreground pixels would penalize more. – eLearner Sep 21 '20 at 09:39
  • Okay I see your point now and thanks for the answer. However, this is not what I actually wanted. I wanted to weight whole image sample rather than a specific class. I guess only solution I can do is to give the same weight of that image samples' pixels and 1 as a weight for all other image samples' pixels. – Burak Bekci Sep 27 '20 at 13:57
  • I see. I would suggest using the following approach for that: https://stackoverflow.com/a/48577360/13319360 – eLearner Sep 28 '20 at 12:13