Multiclass linear SVM in python that return probability

Question

How can I implement a linear SVM for multi-class which returns the proabability matrix for the test samples. Train samples: mxn Train labels: mxc Test labels : mxc, where column has the probability of each class.

The function in sklearn which does "one-vs-the-rest" LinearSVC doesn't return probablity array for each sample like SVC which has predict_proba

Edit

Code:

        print X_train.shape,y.shape
        svc = LinearSVC()
        clf = CalibratedClassifierCV(svc, cv=10)
        clf.fit(X_train, y)

Output:

(7112L, 32L) (7112L, 6L)
Traceback (most recent call last):
  File "SVC_Calibirated_Probability.py", line 171, in <module>
    clf.fit(X_train, y)
  File "C:\Anaconda\lib\site-packages\sklearn\calibration.py", line 110, in fit
    force_all_finite=False)
  File "C:\Anaconda\lib\site-packages\sklearn\utils\validation.py", line 449, in check_X_y
    y = column_or_1d(y, warn=True)
  File "C:\Anaconda\lib\site-packages\sklearn\utils\validation.py", line 485, in column_or_1d
    raise ValueError("bad input shape {0}".format(shape))
ValueError: bad input shape (7112L, 6L)

Answer 1

LinearSVC does not support probability estimates because it is based on liblinear but liblinear supports probability estimates for logistic regression only.

If you just need confidence scores, but these do not have to be probabilities, you can use decision_function instead.

If it it not required to choose penalties and loss functions of linear SVM, you can also use SVC by setting kernel to be 'linear', then you can have predict_proba.

Update #1:

You can use SVC with OneVsRestClassifier to support one-vs-rest scheme, for example

from sklearn import datasets
from sklearn.multiclass import OneVsRestClassifier
from sklearn.svm import SVC
iris = datasets.load_iris()
X, y = iris.data, iris.target
clf = OneVsRestClassifier(SVC(kernel='linear', probability=True, class_weight='auto'))
clf.fit(X, y)
proba = clf.predict_proba(X)

Update #2:

There is another way to estimate probabilities with LinearSVC as classifier.

from sklearn.svm import LinearSVC
from sklearn.calibration import CalibratedClassifierCV
from sklearn.datasets import load_iris

iris = load_iris()
X = iris.data
Y = iris.target
svc = LinearSVC()
clf = CalibratedClassifierCV(svc, cv=10)
clf.fit(X, Y)
proba = clf.predict_proba(X)

However for the other question (Making SVM run faster in python), this solution is not likely to enhance performance either as it involves additional cross-validation and does not support parallelization.

Update #3:

For the second solution, because LinearSVC does not support multilabel classification, so you have to wrap it in OneVsRestClassifier, here is an example:

from sklearn.svm import LinearSVC
from sklearn.calibration import CalibratedClassifierCV
from sklearn.multiclass import OneVsRestClassifier
from sklearn.datasets import make_multilabel_classification

X, Y = make_multilabel_classification(n_classes=2, n_labels=1,
                                      allow_unlabeled=True,
                                      return_indicator=True,
                                      random_state=1)
clf0 = CalibratedClassifierCV(LinearSVC(), cv=10)
clf = OneVsRestClassifier(clf0)
clf.fit(X, Y)
proba = clf.predict_proba(X)