Logistic Regression to support multiple classes directly

Question

My Understanding is Softmax Regression is generalization of Logistic Regression to support multiple classes .

Softmax Regression model first computes a score for each class then estimates the probability of each class by applying the softmax function to the scores.

Each class has its own dedicated parameter vector

My question : Why can't we use Logistic Regression to classify to multiple classes in a much simpler way like if probability is 0 to 0.3 then Class A ; 0.3 to 0.6 then Class B : 0.6 to 0.9 then Class C etc.

Why separate coefficient vector is always needed ?

I'm new to ML . Not sure if this question is due to lack of any fundamental concept understanding .

Answer 1

First up, in terms of terminology, I'd say a more established terminology is multinomial logistic regression.

Softmax function is a natural choice for computing probabilities because it corresponds to MLE. Cross-entropy loss has a probabilistic interpretation as well - that's the "distance" between two distributions (output and target). What you suggest is to discriminate classes in an artificial way - output a binary distribution and somehow compare it to a multi-class distribution. In theory, it is possible and may work, but surely has drawbacks. For example, it is harder to train.

Suppose the output is 0.2 (i.e. class A) and the ground truth is class B. You'd like to tell the network to shift towards a higher value. Next time, the output is 0.7 - the network actually learned and moved in the right direction, but you punish it again. In fact, there are unstable points (0.3 and 0.6 in your example) that the network need time to learn as critical ones. Two values - 0.2999999 and 0.3000001 are almost indistinguishable for the network, but they determine if the result is correct or not.

In general, output as a probability distribution is always better than direct discrimination, because it gives more information.