Confusion matrix for random forest in R Caret

Question

I have data with binary YES/NO Class response. Using following code for running RF model. I have problem in getting confusion matrix result.

 dataR <- read_excel("*:/*.xlsx")
 Train    <- createDataPartition(dataR$Class, p=0.7, list=FALSE)  
 training <- dataR[ Train, ]
 testing  <- dataR[ -Train, ]

model_rf  <- train(  Class~.,  tuneLength=3,  data = training, method = 
"rf",  importance=TRUE,  trControl = trainControl (method = "cv", number = 
5))

Results:

Random Forest 

3006 samples
82 predictor
2 classes: 'NO', 'YES' 

No pre-processing
Resampling: Cross-Validated (5 fold) 
Summary of sample sizes: 2405, 2406, 2405, 2404, 2404 
Addtional sampling using SMOTE

Resampling results across tuning parameters:

 mtry  Accuracy   Kappa    
  2    0.7870921  0.2750655
  44    0.7787721  0.2419762
 87    0.7767760  0.2524898

Accuracy was used to select the optimal model using  the largest value.
The final value used for the model was mtry = 2.

So far fine, but when I run this code:

# Apply threshold of 0.50: p_class
class_log <- ifelse(model_rf[,1] > 0.50, "YES", "NO")

# Create confusion matrix
p <-confusionMatrix(class_log, testing[["Class"]])

##gives the accuracy
p$overall[1]

I get this error:

 Error in model_rf[, 1] : incorrect number of dimensions

I appreciate if you guys can help me to get confusion matrix result.

If you include a [minimal reproducible example](https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) in your question it would be easier to help you. — Samuel, Oct 18 '17 at 18:07

missuse · Accepted Answer · 2019-07-10T10:36:23.687

As I understand you would like to obtain the confusion matrix for cross validation in caret.

For this you need to specify savePredictions in trainControl. If it is set to "final" predictions for the best model are saved. By specifying classProbs = T probabilities for each class will be also saved.

data(iris)
iris_2 <- iris[iris$Species != "setosa",] #make a two class problem
iris_2$Species <- factor(iris_2$Species) #drop levels

library(caret)
model_rf  <- train(Species~., tuneLength = 3, data = iris_2, method = 
                       "rf", importance = TRUE,
                   trControl = trainControl(method = "cv",
                                            number = 5,
                                            savePredictions = "final",
                                            classProbs = T))

Predictions are in:

model_rf$pred

sorted as per CV fols, to sort as in original data frame:

model_rf$pred[order(model_rf$pred$rowIndex),2]

to obtain a confusion matrix:

confusionMatrix(model_rf$pred[order(model_rf$pred$rowIndex),2], iris_2$Species)
#output
Confusion Matrix and Statistics

            Reference
Prediction   versicolor virginica
  versicolor         46         6
  virginica           4        44

               Accuracy : 0.9            
                 95% CI : (0.8238, 0.951)
    No Information Rate : 0.5            
    P-Value [Acc > NIR] : <2e-16         

                  Kappa : 0.8            
 Mcnemar's Test P-Value : 0.7518         

            Sensitivity : 0.9200         
            Specificity : 0.8800         
         Pos Pred Value : 0.8846         
         Neg Pred Value : 0.9167         
             Prevalence : 0.5000         
         Detection Rate : 0.4600         
   Detection Prevalence : 0.5200         
      Balanced Accuracy : 0.9000         

       'Positive' Class : versicolor

In a two class setting often specifying 0.5 as the threshold probability is sub-optimal. The optimal threshold can be found after training by optimizing Kappa or Youden's J statistic (or any other preferred) as a function of the probability. Here is an example:

sapply(1:40/40, function(x){
  versicolor <- model_rf$pred[order(model_rf$pred$rowIndex),4]
  class <- ifelse(versicolor >=x, "versicolor", "virginica")
  mat <- confusionMatrix(class, iris_2$Species)
  kappa <- mat$overall[2]
  res <- data.frame(prob = x, kappa = kappa)
  return(res)
})

Here the highest kappa is not obtained at threshold == 0.5 but at 0.1. This should be used carefully because it can lead to over-fitting.

Thanks. Only one issue, In this code cm pred model only works when I define train as data-set. I think for pred I need to define test data-set. When I code test$Class it gives this error: Error in table(data, reference, dnn = dnn, ...) : all arguments must have the same length — Mike, Oct 19 '17 at 21:12
This code leads to the confusion matrix for cross validation folds in caret. Since cross validation is done on the train set it is good only for the train set. To obtain a confusion matrix on the test set one must first predict the class of the test set samples and compare it to the true class by `confusionMatrix` function. — missuse, Oct 20 '17 at 12:17

score 1 · Answer 2 · answered Oct 18 '17 at 17:48

1

You can try this to create confusion matrix and check accuracy

m <- table(class_log, testing[["Class"]])
m   #confusion table

#Accuracy
(sum(diag(m)))/nrow(testing)

answered Oct 18 '17 at 17:48

Hardik Gupta

4,700
9
41
83

Thank but I get error when running class_log part. I edited my question – Mike Oct 18 '17 at 18:01

score 0 · Answer 3 · answered Oct 18 '17 at 18:01

The code piece class_log <- ifelse(model_rf[,1] > 0.50, "YES", "NO") is an if-else statement that performs the following test:

In the first column of model_rf, if the number is greater than 0.50, return "YES", else return "NO", and save the results in object class_log.

So the code essentially creates a character vector of class labels, "YES" and "NO", based on a numeric vector.

score 0 · Answer 4 · answered Oct 18 '17 at 18:30

0

You need to apply your model to the test set.

prediction.rf <- predict(model_rf, testing, type = "prob")

Then do class_log <- ifelse(prediction.rf > 0.50, "YES", "NO")

answered Oct 18 '17 at 18:30

Victor Kostyuk

621
5
16

thanks. class_log code work for binary Y/N response class? – Mike Oct 18 '17 at 20:14
`prediction.rf` will have real values (note the `type = "prob"`). You can also do `type = "raw"` to get binary right away, but this doesn't let you control the threshold. See `?predict.train` – Victor Kostyuk Oct 18 '17 at 20:21

Confusion matrix for random forest in R Caret

4 Answers4