1

I am using https://www.tensorflow.org/api_docs/python/tf/estimator/DNNClassifier

Let's say I have a Classification problem. Attempting to classify 2 things. Class1 is Happy Face, Class2 is Not Happy Face. In this particular scenario, when looking at 1,000+ samples every day, I just want to grab the top 10 Happy Faces. So really, I just want to be VERY confident that the top 10 faces are Happy Faces. So if it classifies a Happy Face as a Not Happy Face, I'm ok with it. It can miss a few (or quite honestly 100's of the 1000's+ a day it will review). However, if it mis-classifies a Not Happy Face as a Happy Face... I will not be happy with the outcome. 

So we could say that:
GOOD = TP = (truth) Happy Face,      (prediction) Happy Face
BAD  = FP = (truth) Not Happy Face,  (prediction) Happy Face
GOOD = TN = (truth) Not Happy Face,  (prediction) Not Happy Face
OK   = FN = (truth) Happy Face,      (prediction) Not Happy Face

Pretty much, I can live with the occasional FN (False Negative). But I really do NOT want a FP (False Positive)

I am interested in using the weight column to "downweight" all of one class, and "upweight" the other.

In the end, I want some kind of biased loss/cost function. When training, FP should "cost" more, FN, should still cost, but a bit less. Looking at the DNNClassifier, it seems like the weight_column could be the answer.

When training, I could use set all the Happy Faces to a weight of 1.5+ (or something, need to experiment). And I could set all the Not Happy Faces to 0.5 (or whatever).

So, it comes down to a few questions:

  1. Is that what the weight column is intended to be used for?

  2. I see that the weight column is supposed to be a feature/input. In this scenario, I would NOT want the weight column to be used as a feature, because I would assume the model would pick up on the correlation of the weight to the classification, and heavily rely on that input as the predictor. So, does tensorflow use the weight column as an input? I hope it does not.

  3. Assuming/hoping TF does not use the weight column for input, when predicting/evaluating, what doe I pass in for the weight column? 1?

  4. If the weight_column is not the answer, what is the recommended solution to handle a scenario like this (biased cost function for one particular class)? I don't need the model to be 100% accurate on all classifications. But for the one class, when it says it is a Happy Face, I want it to be pretty dang confident that it is a Happy Face. If it misses a few Happy Faces, and it thinks they are Not Happy Faces. It's ok. As long as I get my top 10 "most happy" faces per day, I'm happy :)

Thanks for your help. I appreciate it.

mschmidt42
  • 1,065
  • 1
  • 9
  • 12
  • 1
    Possible duplicate of [Upweight a Category in Tensorflow](https://stackoverflow.com/questions/48098951/upweight-a-category-in-tensorflow) – Maxim Jan 26 '18 at 15:16
  • 1
    All your questions are answered here - https://stackoverflow.com/a/48099304/712995 – Maxim Jan 26 '18 at 15:16
  • Thanks for the link. I've come across that post before. But from the code example, it looks like it is setting all of the weights to "ones". I don't understand how that what help anything. From what I understand, the weights should be set BEFORE training, not during training. Any ideas? – mschmidt42 Jan 26 '18 at 22:31

1 Answers1

0

I think playing with threshold is a better approach to handle FN and FP trade-off. You can do following:

probability_of_1 = predictions from estimator
if probability_of_1 > YOUR_THRESHOLD:
  output = happy-face
else:
  output = not-happy-face
user1454804
  • 1,070
  • 7
  • 6