How to get roc auc for binary classification in sklearn

Question

I have binary classification problem where I want to calculate the roc_auc of the results. For this purpose, I did it in two different ways using sklearn. My code is as follows.

Code 1:

from sklearn.metrics import make_scorer
from sklearn.metrics import roc_auc_score

myscore = make_scorer(roc_auc_score, needs_proba=True)

from sklearn.model_selection import cross_validate
my_value = cross_validate(clf, X, y, cv=10, scoring = myscore)
print(np.mean(my_value['test_score'].tolist()))

I get the output as 0.60.

Code 2:

y_score = cross_val_predict(clf, X, y, cv=k_fold, method="predict_proba")

from sklearn.metrics import roc_curve, auc
fpr = dict()
tpr = dict()
roc_auc = dict()
for i in range(2):
    fpr[i], tpr[i], _ = roc_curve(y, y_score[:,i])
    roc_auc[i] = auc(fpr[i], tpr[i])
print(roc_auc)

I get the output as {0: 0.41, 1: 0.59}.

I am confused since I get two different scores in the two codes. Please let me know why this difference happens and what is the correct way of doing this.

I am happy to provide more details if needed.

What's the value of k_fold? And pls explain what exactly you are trying to do in your code 2. Why exactly should `auc(roc_curve(y, y_score[:,i]))` give a valid AUC value? — desertnaut, Mar 25 '20 at 14:04
@desertnaut thank you for the comment. I see, the k_fold values are different. However, I get bit different values even with the same k_fold. Code1: `0.5864040216803137` and Code2: `{0: 0.41393184645202347, 1: 0.5860681535479765}` :) — EmJ, Mar 25 '20 at 14:04
Please see my updated comment, and ensure that you are actually comparing apples to apples — desertnaut, Mar 25 '20 at 14:05
@desertnaut The source for my second code came from these SO questions: https://stackoverflow.com/questions/43043271/roc-curve-for-binary-classification-in-python, https://stackoverflow.com/questions/45641409/computing-scikit-learn-multiclass-roc-curve-with-cross-validation-cv Please let me know your thoughts. Thank you :) — EmJ, Mar 25 '20 at 14:06
post some data and the full code. In case 1 you use, cv=10 and in case 2 some KFolds. To get the same results. Use the same data, same preprocessing of the data and same sklearn version. — seralouk, Mar 25 '20 at 14:19

seralouk · Accepted Answer · 2020-03-26T08:41:12.870

It seems that you used a part of my code from another answer, so I though to also answer this question.

For a binary classification case, you have 2 classes and one is the positive class.

For example see here. pos_label is the label of the positive class. When pos_label=None, if y_true is in {-1, 1} or {0, 1}, pos_label is set to 1, otherwise an error will be raised..

import matplotlib.pyplot as plt
from sklearn import svm, datasets
from sklearn.metrics import roc_curve, auc
from sklearn.multiclass import OneVsRestClassifier
from sklearn.model_selection import cross_val_predict
from sklearn.linear_model import LogisticRegression
import numpy as np

iris = datasets.load_iris()
X = iris.data
y = iris.target
mask = (y!=2)
y = y[mask]
X = X[mask,:]
print(y)
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]

positive_class = 1

clf = OneVsRestClassifier(LogisticRegression())
y_score = cross_val_predict(clf, X, y, cv=10 , method='predict_proba')

fpr = dict()
tpr = dict()
roc_auc = dict()
fpr[positive_class], tpr[positive_class], _ = roc_curve(y, y_score[:, positive_class])
roc_auc[positive_class] = auc(fpr[positive_class], tpr[positive_class])
print(roc_auc)

{1: 1.0}

and

from sklearn.metrics import make_scorer
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import cross_validate

myscore = make_scorer(roc_auc_score, needs_proba=True)

clf = OneVsRestClassifier(LogisticRegression())
my_value = cross_validate(clf, X, y, cv=10, scoring = myscore)
print(np.mean(my_value['test_score'].tolist()))
1.0

I suspected that the different results in OP's 2nd block are due to different definitions of the positive class, but could not easily demonstrate it — desertnaut, Mar 25 '20 at 14:31
@makis Thanks a lot for the answer. I found it very useful. I have one further question regarding cross validation, that I thought you might have some great ideas to solve it. The question is: https://stackoverflow.com/questions/60851341/how-to-calculate-accuracy-weighted-precision-weighted-recall-weighted-f-measu Please kindly let me know your thoughts on this. Thank you :) — EmJ, Mar 25 '20 at 14:56
@makis I get an error in your 1st code when I set `positive_class = 1`. My `y_bin` looks like follows. `[[1], [0], [0], ..., [1]]`. The error I get is `IndexError: index 1 is out of bounds for axis 1 with size 1`. Just wondering if there is a way to resolve this issue. I look forward to hearing from you. Thank you :) — EmJ, Mar 26 '20 at 00:32

How to get roc auc for binary classification in sklearn

1 Answers1

Linked