1

I am using PyML for SVM classification. However, I noticed that when I evaluate a multi-class classifier using LOO, the results object does not report the sensitivity and PPV values. Instead they are 0.0:

from PyML import *
from PyML.classifiers import multi

mc = multi.OneAgainstRest(SVM())
data = VectorDataSet('iris.data', labelsColumn=-1)
result = mc.loo(data)

result.getSuccessRate()
>>> 0.95333333333333337
result.getPPV()
>>> 0.0
result.getSensitivity()
>>> 0.0

I have looked at the code but couldn't figure out what is going wrong here. Has somebody a workaround for this?

Mathias Loesch
  • 373
  • 1
  • 5
  • 15

2 Answers2

6

You cannot get the usual Precision/Recall measurements on a multi-class problem. You have to get Precision/Recall for each class, and you can compute a weighted average.

I don't know about the specifics of PyML, but you can just go through the predictions and calculate them for each class.

smola
  • 863
  • 8
  • 15
2

For multiclass sensitivity calculation, you can use scikit-learn metrics API.

Notice average=None for sensitivity of each class independently. 

sklearn.metrics.recall_score(Y_true,Y_prediction,average=None)

For instance, if Y has 4 classes, the result will be an array with the sensitivity of each one. 

array([0.96629213, 0.86263736, 0.81920904, 0.7704918])
Juan-Kabbali
  • 1,961
  • 15
  • 20