How to plot AUC ROC for different caret training models?

Question

Here's a reprex

library(caret)
library(dplyr)

set.seed(88, sample.kind = "Rounding")

mtcars <- mtcars %>%
  mutate(am = as.factor(am))

test_index <- createDataPartition(mtcars$am, times = 1, p= 0.2, list = F)

train_cars <- mtcars[-test_index,]

test_cars <- mtcars[test_index,]

set.seed(88, sample.kind = "Rounding")
cars_nb <- train(am ~ mpg + cyl,
                data = train_cars, method = "nb", 
                trControl = trainControl(method = "cv", number = 10, savePredictions = "final"))

cars_glm <- train(am ~ mpg + cyl,
                 data = train_cars, method = "glm", 
                 trControl = trainControl(method = "cv", number = 10, savePredictions = "final"))

My question is, how would I go about creating an AUC ROC curve on a single plot to visually compare the two models?

could this help https://stackoverflow.com/questions/31138751/roc-curve-from-training-data-in-caret — Dr. Flow, May 15 '20 at 08:52
Thanks for the link! the only thing I don't really understand in the answer is why is the parameter-setting set to 2? — bolleke, May 15 '20 at 09:09
mtry: Number of variables randomly sampled as candidates at each split (https://machinelearningmastery.com/tune-machine-learning-algorithms-in-r/) — Dr. Flow, May 15 '20 at 09:16
I tried the MLeval method below, but the nb model doesn't output a probability dataframe from the train function — bolleke, May 15 '20 at 09:50
The first `train` fails with the following error for me: `Error: wrong model type for regression` — Calimo, May 17 '20 at 12:37
... until I load dplyr. Then it works. I edited the question accordingly — Calimo, May 17 '20 at 12:48

Calimo · Accepted Answer · 2020-05-17T14:56:27.750

I assume that you want to show the ROC curves on the test set, unlike in the question pointed in the comment (ROC curve from training data in caret) which uses the training data.

The first thing to do will be to extract predictions on the test data (newdata=test_cars), in the form of probabilities (type="prob"):

predictions_nb <- predict(cars_nb, newdata=test_cars, type="prob")
predictions_glm <- predict(cars_glm, newdata=test_cars, type="prob")

This gives us a data.frame with probabilities to belong to class 0 or 1. Let's use the probability of class 1 only:

predictions_nb <- predict(cars_nb, newdata=test_cars, type="prob")[,"1"]
predictions_glm <- predict(cars_glm, newdata=test_cars, type="prob")[,"1"]

Next I'll use the pROC package to create the ROC curves for the training data (disclaimer: I am the author of this package. There are other ways to achieve the result, but this is the one I am the most familiar with):

library(pROC)
roc_nb <- roc(test_cars$am, predictions_nb)
roc_glm <- roc(test_cars$am, predictions_glm)

Finally you can plot the curves. To have two curves with the pROC package, use the lines function to add the line of the second ROC curve to the plot

plot(roc_nb, col="green")
lines(roc_glm, col="blue")

To make it more readable you can add a legend:

legend("bottomright", col=c("green", "blue"), legend=c("NB", "GLM"), lty=1)

And with the AUC:

legend_nb <- sprintf("NB (AUC: %.2f)", auc(roc_nb))
legend_glm <- sprintf("GLM (AUC: %.2f)", auc(roc_glm))
legend("bottomright",
       col=c("green", "blue"), lty=1,
       legend=c(legend_nb, legend_glm))

Thank you very much for the insightful answer @Calimo, I'm going to use your method instead of the MLeval the evalm function does ask to put a caret training result in. Is there any way one could add the AUC value to the curve? — bolleke, May 17 '20 at 13:16
Please remember to accept this answer if it solved your problem. — Calimo, May 17 '20 at 15:28

How to plot AUC ROC for different caret training models?

1 Answers1