I have a hugely imbalanced data set. To deal with this issue, I tried separately different class-imbalance techniques : downSample, class weights, threshold tuning. Among them, threshold tuning was the least effective. Using downSample alone or class weights alone, I did not manage to get sufficiently good results : either there is too much of FalsePositives or FalseNegatives. So I would like to combine the two techniques. Here is what I tired :
# produce some re-producible imbalanced data
set.seed(12345)
y <- as.factor(sample(c("M", "F"),
prob = c(0.1, 0.9),
size = 10000,
replace = TRUE))
x <- rnorm(10000)
DATA <- data.frame(y = as.factor(y), x)
set.seed(12345)
folds <- createFolds(dataSet$y, k = 10,
list = TRUE, returnTrain = TRUE)
# class weights
k <- 0.5
classWeights <- ifelse(DATA$y == "M",
(1/table(DATA$y)[1]) * k,
(1/table(DATA$y)[2]) * (1-k))
so when I don't put the sampling
argument in controlTrain
:
# select algorithm
algorithm <- "bayesglm"
# train parameters
set.seed(12345)
traincontrol <- trainControl(method = "loocv", # resampling method
number = 10,
index = folds,
classProbs = TRUE,
summaryFunction = twoClassSummary,
savePredictions = TRUE,
# sampling = "down"
)
fitModel <- train(y ~ .,
data = DATA,
trControl = traincontrol,
method = algorithm,
metric = "ROC",
weights = classWeights,
)
it works and there is no error. but when I add the sampling argument of trainControl as
# train parameters
set.seed(12345)
traincontrol <- trainControl(method = "loocv", # resampling method
number = 10,
index = folds,
classProbs = TRUE,
summaryFunction = twoClassSummary,
savePredictions = TRUE,
sampling = "down"
)
fitModel <- train(y ~ .,
data = DATA,
trControl = traincontrol,
method = algorithm,
metric = "ROC",
weights = classWeights,
)
I get this error which is understandable :
Error in model.frame.default(formula = .outcome ~ ., data = list(x = c(-0.0640913631047556, :
variable lengths differ (found for '(weights)')
In addition: There were 11 warnings (use warnings() to see them)
Timing stopped at: 0.112 0.001 0.115
Is there a way to do this in caret
? Many thanks in advance.