3

I am new to semantic segmentation. I used the FCN to train my dataset. In the data set there are some pixels for the unknown class. I would like to exclude this class from my loss. So I defined a weight based on the class distribution of whole dataset and set the weight for the unknown class to zero as following. But I am still getting prediction for this class. Do you have any idea how to properly exclude one specific class? 

loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits
                      (logits=logits, labels=tf.squeeze(annotation,
                                                        squeeze_dims=[3]),name="entropy"))
weighted_losses = (loss * weights)
train_op = optimizer.minimize(weighted_losses,
                                  var_list=tf.trainable_variables(),
                                  global_step=tf.train.get_global_step())

I do not know pytorch, but I heard that there is some thing for this purpose "ignore_index" in loss function and you can ignore a specific class. If this is right approach to my problem, do you know if there is some thing equivalent in tensorflow?

Shai
  • 111,146
  • 38
  • 238
  • 371
Arb
  • 91
  • 10

1 Answers1

6

For semantic segmentation you have 2 "special" labels: the one is "background" (usually 0), and the other one is "ignore" (usually 255 or -1).

  • "Background" is like all other semantic labels meaning "I know this pixel does not belong to any of the semantic categories I am working with". It is important for your model to correctly output "background" whenever applicable.
  • "Ignore" label is not a label that your model can predict - it is "outside" its range. This label only exists in the training annotation meaning "we were unsure how this pixel should be labeled, so just ignore it".

When there are "ignore" pixels in your target labels, your model cannot (and should not) output "ignore" labels. Nevertheless, your model should output something. The fact that this pixel is labeled "ignore" means that whatever your model outputs for that pixel will be ignored by the loss function (assuming you told the loss to ignore "ignore" pixels). Moreover, if your test/validation sets have "ignore" labels means that whatever your model outputs for these pixels, it would simply be ignored by the scoring mechanism and won't be counted as either a correct or incorrect prediction.

To summarize: even when the ground truth has "ignore" labels, the model cannot and should not output "ignore". It simply outputs whatever valid label it feels like and it is perfectly okay.

for tensorflow you can checkout this thread.

Shai
  • 111,146
  • 38
  • 238
  • 371
  • 1
    Thanks for your great answer. I am now wondering how to "ignore" labels in mean iOU. can you give me some hints? – Arb Mar 12 '19 at 08:40
  • @Arb interesting question. why not post it as such? – Shai Mar 12 '19 at 08:53
  • Thanks for reply. I already post it https://stackoverflow.com/questions/55104471/how-to-handle-the-mean-intersection-over-union-miou-for-unknown-class-in-seman – Arb Mar 12 '19 at 09:00