1

Anybody know how to extract the standard deviation for a ResampleResult/BenchmarkResult in mlr3? The implemented metrics seems to be returning only the average value.

measures <- list(
  mlr3::msr("classif.fbeta", predict_sets = "train", id = "fbeta_train"),
  mlr3::msr("classif.fbeta", id = "fbeta_test")
)
bmr$aggregate(measures)
Lucas Teixeira
  • 41
  • 1

1 Answers1

1

Calculating standard errors from Resampling Results is not as straight-forward, as e.g. the folds of a cross-validation are not independent.

We are currently working on integrating some methods to calculate standard errors and confidence intervals, as well as providing new resampling techniques that allow for proper inference of the generalization error.

Sebastian
  • 865
  • 5
  • 13
  • Thank you for the fast answer. I have been following the discussion on mlr3benchmark regarding SE and CI. I was hoping to find an easy way to compute only SD for the outer resampling only, usually journals ask for any variance metric when performing CV. – Lucas Teixeira Feb 03 '23 at 18:44
  • The latest discussions are not being held in the package yet. You can e.g. read up on Bayle (2020) Cross-validation Confidence Intervals for Test Error, which I think gives 2 variance estimators that you would have to implement manually. (We will soon implement methods like this in mlr3 as well) – Sebastian Feb 04 '23 at 08:57