0

I have a multi-class classification task. I can obtain accuracy and balanced accuracy metrics from sklearn in Python but they both spew one figure. How can I obtain balanced accuracy for every class as well? Note that, when I use the caret package in R for modelling, with the metrics report I get balanced accuracy for every class (e.g., the last row in the below example), and I'd like to have the same functionality in sklearn or any accompanying Python library if possible.

Example in caret metrics in R

confusionMatrix(testSet$classes,testSet$pred_model)

output:


                     Class: A   Class: B   Class: C  Class: D  Class: E  Class: F
Sensitivity            0.37143   0.23404   0.25490   0.15254   0.30909   0.27692
Specificity            0.85921   0.84528   0.85057   0.83004   0.86381   0.86235
Pos Pred Value         0.25000   0.21154   0.25000   0.17308   0.32692   0.34615
Neg Pred Value         0.91538   0.86154   0.85385   0.80769   0.85385   0.81923
Prevalence             0.11218   0.15064   0.16346   0.18910   0.17628   0.20833
Detection Rate         0.04167   0.03526   0.04167   0.02885   0.05449   0.05769
Detection Prevalence   0.16667   0.16667   0.16667   0.16667   0.16667   0.16667
Balanced Accuracy      0.61532   0.53966   0.55274   0.49129   0.58645   0.56964

Example in sklearn in Python

acc=accuracy_score(y_test,y_pred)

0.52345

bal_acc=balanced_accuracy_score(y_test,y_pred)

0.53657

Thanks!

Maryam Nasseri
  • 67
  • 7

2 Answers2

1

By sklearn's definition, accuracy and balanced accuracy are only defined on the entire dataset. But you can get per-class recall, precision and F1 score from sklearn.metrics.classification_report.

Matt Hall
  • 7,614
  • 1
  • 23
  • 36
  • Thanks for your comment. I have already obtained other metrics per class as you have pointed out. I was looking for accuracy scores per class as well. Do you know of any Python libraries or stand-alone functions that can work with the output of sklearn and give accuracy per class? Are there any other libraries other than sklearn that can do ensemble modelling (stacking) and give accuracy per class? Thanks. – Maryam Nasseri Apr 13 '23 at 11:13
  • 1
    If I were you I'd compute them myself, eg [like this](https://stackoverflow.com/a/65673016/20623798), then you know exactly what you're getting. – Matt Hall Apr 13 '23 at 13:35
0

Below is my own answer for the record and for anyone who will stumble upon this question. Matt Hall in the above comment suggested a link to get the accuracy scores from the confusion matrix as matrix.diagonal()/matrix.sum(axis=1). Upon studying that link and similar links, I found out that this actually calculates recall and if axis is set to 0, it gives precision.

The correct way in Python is to extract True Positives, True Negatives, False Positives, and False negatives from the confusion matrix and then use the formula of accuracy below to get the accuracies per class:

# Overall accuracy
ACC = (TP+TN)/(TP+FP+FN+TN)

But again this gives an overall accuracy. If you need balanced accuracy scores per class as in the example I showed above from the Caret package in R, the best way is to export your confusion matrix from the sklearn model (in form of a concatenate) in R, define rownames and colnames, and then use the Caret package to calculate balanced accuracies:

confusionMatrix(t(matrix), mode = "everything")
Maryam Nasseri
  • 67
  • 7