0

I've been using caret to build a decision tree model using the "rpart2" method and 10-fold cross-validation repeated 5 times in R.

set.seed(888)
DT_up_rpart <-train(DiagDM1~., data = training, method = "rpart2", trControl = ctrl2,metric = "ROC", tuneGrid = maxdepthGrid)

From this call I get this results:

## CART
## 56662 samples
##    11 predictor
##     2 classes:'No','Si'
## No pre-processing
## Resampling: Cross-Validated (10 fold, repeated 5 times)
## Summary of sample sizes: 50995, 50995, 50996, 50996, 50996, 50996, ...
## Resampling results across tuning parameters:
##
##   maxdepth  ROC        Sens       Spec       Accuracy   Kappa
##   3         0.7378576  0.7408495  0.6449825  0.6929158  0.3858318
##   4         0.7382515  0.7367128  0.6502273  0.6934699  0.3869400
##   5         0.7382515  0.7367128  0.6502273  0.6934699  0.3869400
##   6         0.7382515  0.7367128  0.6502273  0.6934699  0.3869400
##
## ROC was used to select the optimal model using  the largest value.
## The final value used for the model was maxdepth = 4

As you can see the final value used for the model was maxdepth = 4. So, now using the same training dataset, the same seed, the same performance measures, etc, I train a new model using maxdepth = 4 (constant value):

set.seed(888)
DT_up_rpart2 <-train(DiagDM1~., data = training, method = "rpart2", trControl = ctrl2, metric = "ROC", tuneGrid =data.frame(maxdepth = 4))

And I got these results:

## Resampling: Cross-Validated (10 fold, repeated 5 times)## Summary of sample sizes: 50995, 50995, 50996, 50996, 50996, 50996, ...
## Resampling results:
##   ROC        Sens       Spec       Accuracy   Kappa
##   0.7379012  0.7391975  0.6464791  0.6928382  0.3856765
##
## Tuning parameter'maxdepth'was held constant at a value of 4

Note that in the first results showed above I got ROC = 0.7382512 for maxdepth = 4 while in the second model I got ROC = 0.7379012 for maxdepth = 4, and the same happens for all the performance measures. I was expecting to obtain the same results from the two models. That's why I set the seed value in both cases.

Why am I getting different results? I'm using the same dataset, the same seed, etc... Or am I defining the seed in the wrong way?

Any help would be appreciated.

phiver
  • 23,048
  • 14
  • 44
  • 56
Gerardo Felix
  • 25
  • 7
  • this is just a random guess as I never analyzed this library - but when you develop R libraries you actually code them in C, and you can simply ignore R random seed state, maybe caret is doing so? Have you tried to run the last line multiple times? If you set seed to 888 and run with depth 4, it changes every time? if so - the seeding is the problem, if not - something else. – lejlot May 09 '16 at 23:34
  • Thanks for your suggestion @lejilot. I ran the last code several times setting the seed to 888 and I got the same results every time. I also tried defining the "seeds" argument in the trainControl function as described [here](http://stackoverflow.com/questions/13403427/fully-reproducible-parallel-models-using-caret) but it didn't work. Still getting different results... – Gerardo Felix May 15 '16 at 00:17

0 Answers0