4

I have a hugely imbalanced data set. To deal with this issue, I tried separately different class-imbalance techniques : downSample, class weights, threshold tuning. Among them, threshold tuning was the least effective. Using downSample alone or class weights alone, I did not manage to get sufficiently good results : either there is too much of FalsePositives or FalseNegatives. So I would like to combine the two techniques. Here is what I tired :

# produce some re-producible imbalanced data
set.seed(12345)
y <- as.factor(sample(c("M", "F"),
                      prob = c(0.1, 0.9),
                      size = 10000,
                      replace = TRUE))


x <- rnorm(10000)


DATA <- data.frame(y = as.factor(y), x)

set.seed(12345)
folds <- createFolds(dataSet$y, k = 10, 
                     list = TRUE, returnTrain = TRUE)

# class weights 
k <- 0.5
classWeights <- ifelse(DATA$y == "M",
                       (1/table(DATA$y)[1]) * k,
                       (1/table(DATA$y)[2]) * (1-k))

so when I don't put the sampling argument in controlTrain :

# select algorithm
algorithm <- "bayesglm"

# train parameters
set.seed(12345)
traincontrol <- trainControl(method = "loocv", # resampling method
                             number = 10,
                             index = folds,
                             classProbs = TRUE, 
                             summaryFunction = twoClassSummary,
                             savePredictions = TRUE,
                             # sampling = "down"
                             )

fitModel <- train(y ~ .,
                  data = DATA, 
                  trControl = traincontrol,
                  method = algorithm,
                  metric = "ROC",
                  weights = classWeights,
                  )

it works and there is no error. but when I add the sampling argument of trainControl as

# train parameters
set.seed(12345)
traincontrol <- trainControl(method = "loocv", # resampling method
                             number = 10,
                             index = folds,
                             classProbs = TRUE, 
                             summaryFunction = twoClassSummary,
                             savePredictions = TRUE,
                             sampling = "down"
                             )

fitModel <- train(y ~ .,
                  data = DATA, 
                  trControl = traincontrol,
                  method = algorithm,
                  metric = "ROC",
                  weights = classWeights,
                  )

I get this error which is understandable :

Error in model.frame.default(formula = .outcome ~ ., data = list(x = c(-0.0640913631047556,  : 
  variable lengths differ (found for '(weights)')
In addition: There were 11 warnings (use warnings() to see them)
Timing stopped at: 0.112 0.001 0.115

Is there a way to do this in caret ? Many thanks in advance.

Basilique
  • 150
  • 1
  • 11

0 Answers0