How can I get a confusion matrix of a single run in sklearn cross_validate?

Question

When I do something like:

scoring = ["accuracy", "balanced_accuracy", "f1", "precision", "recall", "roc_auc"]

scores = cross_validate(SVC(), my_x, my_y, scoring = scoring , cv=5, verbose=3, return_train_score=True, return_estimator=True)

how can I get a confusion matrix of a single validation run, e.g. the first one or ideally the best one?

I don't need a plot or something beautiful, only the numbers. If I could see at least the split, then I could recalculate it.

Does this answer your question? [How to write a confusion matrix in Python?](https://stackoverflow.com/questions/2148543/how-to-write-a-confusion-matrix-in-python) — Trenton McKinney, Jun 24 '20 at 19:12
Use grid_search to find optimal model then do matrix off of that. — wolf7687, Jun 24 '20 at 19:18

afsharov · Answer 1 · 2020-06-24T21:43:22.733

If you want to use cross-validation to perform something quite specific during each iteration, maybe it is best to use a CV splitter like StratifiedKFold :

from sklearn.metrics import accuracy_score, confusion_matrix
from sklearn.model_selection import StratifiedKFold
from sklearn.svm import SVC

svm = SVC()
kf = StratifiedKFold(n_splits=5)

scores = []
results = []
for train_index, test_index in kf.split(X, y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    svm.fit(X_train, y_train)
    y_pred = svm.predict(y_test)
    scores.append(accuracy_score(y_test, y_pred)) # use other scoring as prefered 
    results.append(confusion_matrix(y_test, y_pred))

This will compute the confusion matrix for each of the five iterations and store them in results. If you want to get the confusion matrix of the best validation round, you can additionally compute the scoring metric in the loop as well (see the scores list) and retrieve the corresponding confusion matrix.

How can I get a confusion matrix of a single run in sklearn cross_validate?

1 Answers1