1

I am transferring over from caret to the tidymodels environment and I am having an issue with transferring a custom metric. All the available documentation I can find provides examples of building a custom metric using the 'truth' and 'estimate' variables. Accuracy and roc_auc (using truth and estimate) aren't appropriate as the problem involves betting and odds, so the success isn't comparable across events as the odds make one success more favourable than another and I would like to optimise on this.

I have posted to RStudio community with some code (link below) but no success yet, but I guess the question I have is; does anyone know if this is even possible in yardstick currently?

It may be that I simply have to go back to caret.

Thanks Chris

https://community.rstudio.com/t/tidymodels-custom-metric-for-multi-class-classification-yardstick-machine-learning/122648?u=nealec

nealec
  • 21
  • 3
  • Curious if you got anywhere on this? I posted something similar: https://github.com/tidymodels/yardstick/issues/220 – Bryan Shalloway May 13 '22 at 19:05
  • I'm afraid not. There is the ability to build a custom metric but I found it rather convoluted and in the end returned to caret. – nealec May 15 '22 at 08:12
  • Gotcha, I opened another related issue where I link to a few other issues where people made custom metrics: https://github.com/tidymodels/yardstick/issues/305 -- in case those may be helpful. Though I still haven't seen any examples where a custom metric is being built for a numeric metric that takes in multiple arguments for estimate. – Bryan Shalloway May 16 '22 at 19:17

0 Answers0