How to present accuracy of different models using caret package in the same list

Question

I'm trying to test models performances using the caret package.I got the results per each of the models but I wouldlike to get a list that will contain the accuracy and ROC of all the models together.How can I do it? Here is my toy data and two models:

dat <- read.table(text = " target birds    wolfs     snakes
        0        3        9         7
        1        3        8         4
        1        1        2         8
        0        1        2         3
        0        1        8         3
        1        6        1         2
        0        6        7         1
        1        6        1         5
        0        5        9         7
        1        3        8         7
        1        4        2         7
        0        1        2         3
        0        7        6         3
        1        6        1         1
        0        6        3         9
        1        6        1         1   ",header = TRUE)

Here are the two models:

svmRadial <- train(target ~ ., data = dat, method='svmRadial')
glm <- train(target ~ ., data = dat, method='glm')

I would like to get such a table an an output:

ModelName  Accuracy  ROC
svmRadial   0.95     0.74
glm         0.93     0.7

score 4 · Accepted Answer · edited May 23 '17 at 11:50

This is essentially a question on customizing the summaryFunction. You can see a similar question here. Here is a function that is a combination of the defaultSummary and twoClassSummary functions.

mySummary <- function(data, lev = NULL, model = NULL)
{
    requireNamespace("pROC")
    if (!all(levels(data[, "pred"]) == levels(data[, "obs"]))) 
        stop("levels of observed and predicted data do not match")
    rocObject <- try(pROC::roc.default(data$obs, data[, lev[1]]), 
                     silent = TRUE)
    rocAUC <- if (class(rocObject)[1] == "try-error"){ 
        NA
    }else{rocObject$auc}

    if (!is.factor(data$obs)) 
        data$obs <- factor(data$obs, levels = lev)
    Acc <- postResample(data[, "pred"], data[, "obs"])[1]

    out <- c(Acc, rocAUC)
    names(out) <- c("Accuracy","ROC")
    out
}


fitControl <- trainControl(classProbs = TRUE,
                           summaryFunction = mySummary)

set.seed(123)
svmRadial_acc_roc <- train(as.factor(target) ~ ., data = dat, method='svmRadial', trControl=fitControl)
glm_acc_roc <- train(as.factor(target) ~ ., data = dat, method='glm', trControl=fitControl)

I believe it is considered better practice to look at the distribution of the results. To do so, you would use the resamples function.

results <- resamples(list(svm=svmRadial_acc_roc, glm=glm_acc_roc))
summary(results)

Call:
summary.resamples(object = results)

Models: svm, glm 
Number of resamples: 25 

Accuracy 
      Min. 1st Qu. Median   Mean 3rd Qu.   Max. NA's
svm 0.2500  0.5000  0.625 0.6034  0.6667 1.0000    0
glm 0.1667  0.4286  0.500 0.4993  0.6000 0.7143    0

ROC 
      Min. 1st Qu. Median   Mean 3rd Qu. Max. NA's
svm 0.4444  0.5608 0.6667 0.7422     1.0    1    1
glm 0.4444  0.6250 0.6667 0.7108     0.8    1    0

That said, if you really want that simple table.

# svm had some cross-validation so pull 'best tune'
svm_result <- svmRadial_acc_roc$results[
    svmRadial_acc_roc$results$C == svmRadial_acc_roc$bestTune$C,
    c("Accuracy", "ROC")]
glm_result <- glm_acc_roc$results[,c("Accuracy", "ROC")]

# make data.frame
data.frame(ModelName = c("svmRadial", "glm"),
           Accuracy = c(svm_result$Accuracy, glm_result$Accuracy),
           ROC = c(svm_result$ROC, glm_result$ROC)
)

  ModelName  Accuracy       ROC
1 svmRadial 0.6034444 0.7421875
2       glm 0.4993333 0.7107778

There is also a function called `getTrainPerf` that will extract the resampled performance estimates for the optimal parameters. — topepo, Feb 15 '15 at 22:28

How to present accuracy of different models using caret package in the same list

1 Answers1