2

When I train to models, one using classProbs=TRUE and the other without computing the probabilities, I get different results:

library(caret)

set.seed(7)
myControl <- trainControl(method='cv',number=3,savePredictions=TRUE)
set.seed(7)
model <- train(Species~., iris, tuneLength=4,method='svmRadial', trControl=myControl)

set.seed(7)
myControl <- trainControl(method='cv',number=3,savePredictions=TRUE,classProbs=TRUE)
set.seed(7)
modelProbs <- train(Species~., iris, tuneLength=4, method='svmRadial', trControl=myControl)

The model with classProbs=FALSE (var model) is:

C     Accuracy   Kappa      Accuracy SD  Kappa SD  
0.25  0.9266667  0.8900100  0.03055050   0.04584410
0.50  0.9333333  0.8999840  0.02309401   0.03469643
1.00  0.9400000  0.9099880  0.02000000   0.03004800
2.00  0.9400000  0.9100059  0.02000000   0.02999416

And the model with classProbs=TRUE (var modelProb) is:

C     Accuracy   Kappa     Accuracy SD  Kappa SD  
0.25  0.9266667  0.890010  0.03055050   0.04584410
0.50  0.9333333  0.899984  0.02309401   0.03469643
1.00  0.9400000  0.909988  0.02000000   0.03004800
2.00  0.9466667  0.919980  0.02309401   0.03466529

resulting even in different final models after parameter selection (C=1 when classProbs=FALSE and C=2 when classProbs=TRUE).

I have found out that all predictions are equal for both models except those where the classifier is not very sure of which class to predict. For example:

> model$pred[423,]
          pred        obs rowIndex     sigma    C Resample
423 versicolor versicolor       69 0.8071298 0.25    Fold3

> modelProbs$pred[423,]
         pred        obs     setosa versicolor virginica rowIndex        sigma    C Resample
423 virginica versicolor 0.03307154  0.4813102 0.4856182       69 0.8071298 0.25    Fold3

In this experiment, differences are very small but I have tried with more complex data, and differences are huge. Can anybody explain how the classProbs attribute affects to predictions? I thought it was used to view the probabilities for each class but that it didn't change the results.

pglez82
  • 489
  • 5
  • 11
  • In your post, results are the same. Please revise. –  Apr 21 '15 at 08:48
  • Actually it is done by the kernlab package and the package documents explains as 'In classification when prob.model is TRUE a 3-fold cross validation is performed on the data and a sigmoid function is fitted on the resulting decision values f.'. I guess it'd be the source of discrepancy - page 57 http://cran.r-project.org/web/packages/kernlab/kernlab.pdf – Jaehyeon Kim Apr 21 '15 at 10:50
  • Thanks for your response @JaehyeonKim. Doing more research I have found this post that answers my question [http://stackoverflow.com/questions/15503027/why-are-probabilities-and-response-in-ksvm-in-r-not-consistent]. The problem is that in my real experiment, which has 121 classes, using the model without probs I get 50% accuracy with a standard 10 folds cv, but when the probs are computed, it drops down to less than 15%! – pglez82 Apr 22 '15 at 09:35

0 Answers0