5

I understand that binary cross-entropy is the same as categorical cross-entropy in case of two classes.

Further, it is clear for me what softmax is.
Therefore, I see that categorical cross-entropy just penalizes the one component (probability) that should be 1.

But why, can't or shouldn't I use binary cross-entropy on a one-hot vector?

Normal Case for 1-Label-Multiclass-Mutual-exclusivity-classification:
################
pred            = [0.1 0.3 0.2 0.4]
label (one hot) = [0   1   0   0]
costfunction: categorical crossentropy 
                            = sum(label * -log(pred)) //just consider the 1-label
                            = 0.523
Why not that?
################
pred            = [0.1 0.3 0.2 0.4]
label (one hot) = [0   1   0   0]
costfunction: binary crossentropy
                            = sum(- label * log(pred) - (1 - label) * log(1 - pred))
                            = 1*-log(0.3)-log(1-0.1)-log(1-0.2)-log(1-0.4)
                            = 0.887

I see that in binary cross-entropy the zero is a target class, and corresponds to the following one-hot encoding:

target class zero 0 -> [1 0]
target class one  1 -> [0 1]

In summary: Why do we just calculate/summarize the negative log likelihood for the predicted class. Why don't we penalize the other SHOULD-BE-ZERO-/NOT-THAT-CLASS classes?

In case one uses binary cross-entropy to a one-hot vector. Probabilities to expected zero labels would be penalized too.

Maxim
  • 52,561
  • 27
  • 155
  • 209
hallo02
  • 307
  • 3
  • 11

1 Answers1

5

See my answer on a similar question. In short, binary cross-entropy formula doesn't make sense for the one-hot vector. It's either possible to apply softmax cross-entropy for two or more classes or use the vector of (independent) probabilities in label, depending on the task.

But why, can't or shouldn't I use binary crossentropy on a one-hot vector?

What you compute is binary cross-entropy of 4 independent features:

pred   = [0.1 0.3 0.2 0.4]
label  = [0   1   0   0]

The model inference predicted that first feature is on with 10% probability, the second feature is on with 30% probability and so on. Target label is interpreted this way: all features are off, except for the second one. Note that [1, 1, 1, 1] is a perfectly valid label as well, i.e. it's not one-hot vector, and pred=[0.5, 0.8, 0.7, 0.1] is a valid prediction, i.e. the sum doesn't have to equal to one.

In other words, your computation is valid, but for a completely different problem: multi-label non-exclusive binary classification.

See also the difference between softmax and sigmoid cross-entropy loss functions in tensorflow.

Maxim
  • 52,561
  • 27
  • 155
  • 209