0

I have a multiclass SVM classifier with labels 'A', 'B', 'C', 'D'.

This is the code I'm running:

>>>print clf.predict([predict_this])
['A']
>>>print clf.decision_function([predict_this])
[[ 185.23220833   43.62763596  180.83305074  -93.58628288   62.51448055  173.43335293]]

How can I use the output of decision function to predict the class (A/B/C/D) with the highest probability and if possible, it's value? I have visited https://stackoverflow.com/a/20114601/7760998 but it is for binary classifiers and could not find a good resource which explains the output of decision_function for multiclass classifiers with shape ovo (one-vs-one).

Edit:

The above example is for class 'A'. For another input the classifier predicted 'C' and gave the following result in decision_function

[[ 96.42193513 -11.13296606 111.47424538 -88.5356536 44.29272494 141.0069203 ]]

For another different input which the classifier predicted as 'C' gave the following result from decision_function,

[[ 290.54180354 -133.93467605  116.37068951 -392.32251314 -130.84421412   284.87653043]]

Had it been ovr (one-vs-rest), it would become easier by selecting the one with higher value, but in ovo (one-vs-one) there are (n * (n - 1)) / 2 values in the resulting list.

How to deduce which class would be selected based on the decision function?

Community
  • 1
  • 1
Samkit Jain
  • 1,560
  • 2
  • 16
  • 33
  • (1) Why you want to do this yourself instead of sklearn doing it for you (```probability=True```, then call ```clf.predict_proba```)? (2) Without extra-steps (which might be costly) as sklearn's ```probability=True``` is doing, you can't get probabilities out of this decision-function. It's not natural for SVMs (sklearn's svm-implementation internally uses [platt-scaling](https://en.wikipedia.org/wiki/Platt_scaling)). – sascha Apr 15 '17 at 11:52
  • @sascha I want to avoid running the classifier again. It is very slow. I have saved the classifier to disk using pickle. Anyway I can get the desired result using probability=True without running it again? – Samkit Jain Apr 15 '17 at 12:27
  • No. These calcs are during training-time. – sascha Apr 15 '17 at 12:36
  • Those values are probably just the pairs resulting from lexicographic-ordering (0/1, 0/2, 0/3, 1/2, 1/3, 2/3) but it won't matter. As i said before, you would need much more work to get probabilities. And this will be more trouble than retraining, guaranteed. – sascha Apr 15 '17 at 13:04

2 Answers2

0

If I understood it right, I think this is a way to do it.I used np.argmax() which returns the index of the highest score.

import numpy as np
a = clf.decision_function([predict_this])
index = np.argmax(a)
print str(unichr(65 + index))

EDIT: Yes, this is not the way it should go, sorry I did not pay attention that you have used decision function. Well the predict function uses the decision function output to calculate the final output, no definite algorithm is given in scikit-learn documentation but if you still want to predict yourself using decision function, see the second comment in the link you provided. That link has mathematical derivation for transforming decision into a prediction.

Siddhant
  • 66
  • 1
  • 9
  • This is not the right way. It would have worked if the number of items in the array were equal to number of classes. Here, I have #classes = 4 and #itemsInArray = 6. We have #itemsInArray = (#classes * (#classes - 1)) / 2 – Samkit Jain Apr 15 '17 at 09:26
  • (1) This can never generate (good) probabilities (2) The link also shows a well-received comment, that the multi-class case is different! – sascha Apr 15 '17 at 13:02
  • FYI, I mean the second post, not the first one. – Siddhant Apr 15 '17 at 15:12
0

For anyone who is looking for an answer to this question, please visit this accepted answer I had posted the same question on DataScience Stack Exchange and got the required answer.

Samkit Jain
  • 1,560
  • 2
  • 16
  • 33