5

Task is to determine which of 3 classes does an image belongs to, or none.

I received a ready model. EfficientNet B4 with ImageNet weights had transfer learning applied to identify 4 classes: 3 target ones and a 4th "None". Latter was trained on examples of random images not containing any of target objects.

Question is if it the correct approach – is the 4th class needed?

My intuition is that net should be trained on the 3 target classes only. Should the output probabilities stay below some threshold (90%?), image should be considered as NOT containing any of the target objects. Am I right?

Serge
  • 1,531
  • 2
  • 21
  • 44

1 Answers1

7

Due to the nature of the softmax function and the manner in which the network is trained, you need the 4th class.

Let's see a concrete example: You train your network to distinguish between apples, oranges and bananas. However, you somehow get the photo of a plum.

You might be surprised at first sight but you need the other class in your dataset. There is no guarantee that using a thresholding will help you eliminate the other class.

You may expect the following two cases:

  1. The output probability is guaranteed to be 1/N for an unknown class, given that you are testing on an unknown N+1 class.
  2. A certain threshold beyond which (like you assumed) < 90% it is not class.

Assume the next cases:

  1. What if you have a case in which an apple really looks like an orange, and your model correctly predicts 40% apple, 30% orange, 30% banana, but since you applied your threshold a correctly identified apple (True Positive) is eliminated? A simple case in which you eliminate the good output of your network
  2. You can still have a 91% assignation to a class, although the new 'fruit' arrival is not part of your dataset; this is due to the inherent calculations and the manner in which softmax works.

Personal Experience: I have once trained a network to distinguish between many types of traffic signs. Out of pure curiosity, I gave it an example of one living room chairs. I expected the same thing like you(the thresholding), but much to my surprise, it was 85% "Yield Way".

Timbus Calin
  • 13,809
  • 5
  • 41
  • 59
  • Thanks for sharing Your experience! Selection of "negative" images to train on appears to be quite important – not just random images? ImageNet, initially used to train the net, as I read somewhere, represents the "knowledge" of the entire "world of images". Thus adding negative examples won't bring anything radically new? – Serge Feb 21 '20 at 15:53
  • 1
    Well, the main difference lies in the fact that by means of transfer learning, a lot of pre-trained features are transferred and thus your learning problem is simplified a lot. Moreover, you can also achieve better results/greater accuracy by means of transfer learning. However that does not mean that adding negative examples will not bring important changes: you still need to train the problem on your own dataset(on those n classes). Again, you might just hope that if a lot of features are transferred since the dataset is big and diverse it will eliminate any possible problem(splitthecomment) – Timbus Calin Feb 21 '20 at 16:03
  • 1
    The idea is that it will not eliminate any problem, but it will ease the actual problem you want to solve(by means of transfer learning the training progresses faster and you can also achieve better results). Eventually, if you want to differentiate between, say apples, oranges, plums and 'other fruits', you still need to create the fourth class named 'other_fruit' which will contain numerous examples of many fruits. – Timbus Calin Feb 21 '20 at 16:05