Using Vetiver in R, why am I getting a metric_set error when trying to run compute_metrics() for a classification model?

Question

I am relatively new to using Vetiver with RStudio and I am trying to plot some accuracy metrics. I am using the metrics: accuracy and kap but when I pass in all the parameters into compute_metrics I get this error, Error Message.

I ran the demo code on the Vetiver site and was following a similar procedure modifying it for a classification model instead of a regression.

Here is my code below and the original data trained on the vetiver model doesn't have a timestamp. It was added for newData1 which is passed into my vetiver model for monitoring. The timestamps are produced to simulate adding new data to the model over time.

library(parsnip)
library(recipes)
library(workflows)
library(tidyverse)
data(bivariate, package = "modeldata")
bivariate_train

biv_rec <-
  recipe(Class ~ ., data = bivariate_train) %>%
  step_BoxCox(all_predictors())%>%
  step_normalize(all_predictors())

svm_spec <-
  svm_linear(mode = "classification") %>%
  set_engine("LiblineaR")

svm_fit <- 
  workflow(biv_rec, svm_spec) %>%
  fit(sample_frac(bivariate_train, 0.7))

library(vetiver)

v <- vetiver_model(svm_fit, "biv_svm")
v

library(pins)
model_board <- board_temp(versioned = TRUE)
model_board %>% vetiver_pin_write(v)

svm_fit <- 
  workflow(biv_rec, svm_spec) %>%
  fit(sample_frac(bivariate_train, 0.7))

v <- vetiver_model(svm_fit, "biv_svm")

model_board %>% vetiver_pin_write(v)

model_board %>% pin_versions("biv_svm")

library(plumber)
pr() %>%
  vetiver_api(v)

vetiver_write_plumber(model_board, "biv_svm")

bivariate_val

#Add datestamp to validation data
values = seq(from = as.Date("2021-01-01"), to = as.Date("2021-10-27"), by = 'day')
# ValDate <- merge(bivariate_val, values)
bivariate_val$date_obs <- values

newData1 <- bivariate_val[1:150,]
Data2 <- bivariate_val[151:300,]

class_metrics_T <- metric_set(accuracy, kap)
original_metrics <-
  augment(v,new_data = newData1) %>%
  vetiver_compute_metrics(date_obs,"week",Class,class_metrics_T)

This is a sample of newData1, I am trying to predict the class as either one or two based on 2 features A,B. Dataset to add for computing metrics

Also don't know if this adds any value, but my class column for predicting is of type factor(fct)

score 0 · Answer 1 · answered Dec 10 '22 at 01:02

I made a bit of a smaller example to show you what may have gone wrong:

library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(tidymodels)
data(bivariate, package = "modeldata")
monitoring <- bivariate_val %>% mutate(date = lubridate::today() + 299)
glm_fit <- 
    workflow(
        Class ~ .,
        logistic_reg()
    ) %>% fit(bivariate_train)

library(vetiver)
#> 
#> Attaching package: 'vetiver'
#> The following object is masked from 'package:tune':
#> 
#>     load_pkgs
v <- vetiver_model(glm_fit, "two-classes")

augment(v, new_data = monitoring) %>%
    vetiver_compute_metrics(
        date, 
        "week", 
        Class, 
        .pred_class, 
        metric_set = yardstick::metric_set(accuracy, kap)
    )
#> # A tibble: 2 × 5
#>   .index        .n .metric  .estimator .estimate
#>   <date>     <int> <chr>    <chr>          <dbl>
#> 1 2023-10-04   300 accuracy binary         0.747
#> 2 2023-10-04   300 kap      binary         0.415

^{Created on 2022-12-09 with reprex v2.0.2}

When you call vetiver_compute_metrics() you need to pass in both truth and estimate. I think for your example it would be Class and .pred_class, the same as mine.

Using Vetiver in R, why am I getting a metric_set error when trying to run compute_metrics() for a classification model?

1 Answers1