Log (and then apply) Spark MLlib model from R to MLflow

Question

I'm using Spark MLlib functions (through the sparklyr package) to train a model but now seem unable to save the model in MLflow for future use.

iris_tbl <- sparklyr::copy_to(sc, iris, "iris_spark")
mdl_mllib <- iris_tbl %>% sparklyr::ml_linear_regression(formula = Sepal_Width ~ Sepal_Length)
mlflow::mlflow_log_model(mdl_mllib, "artifact_path_where_saved")
Error in UseMethod("mlflow_save_model") : 
  no applicable method for 'mlflow_save_model' applied to an object of class "c('ml_model_linear_regression', 'ml_model_regression', 'ml_model_prediction', 'ml_model')"

packageVersion("mlflow")
[1] ‘1.17.0’

What is a simple way to save this model in mlflow for later use on a Spark DataFrame such as:

mlflow::mlflow_load_model(model_uri = "models:/mdl_mllib_project01/Staging")

For context, I'm using Azure Databricks as the ecosystem.

Other places I've looked for answers

These links don't seem to directly solve my problem (https://www.mlflow.org/docs/latest/models.html#id28; https://www.mlflow.org/docs/latest/python_api/mlflow.spark.html#mlflow.spark.log_model), although maybe it's that I need to make a "pipeline" or that this works for Python but not yet for R?
This question seems relevant but offers abstract answers rather than details (How to serve a Spark MLlib model?).

Hello! I am trying to do the same thing and have the same problem. I'm using spark MLlib and am trying to save a model in MLflow. Were you able to find a solution to this? Thank you! — piper180, Feb 23 '22 at 18:39
@ava Sorry, I ended up making an H2O model instead, which I can save and use later through `mlflow`. :( — josephD, Mar 11 '22 at 17:37
No worries! Thanks for letting me know! Would you mind sharing the H20 documentation page that works with sparklyr? — piper180, Mar 14 '22 at 19:48

Log (and then apply) Spark MLlib model from R to MLflow

Other places I've looked for answers

0 Answers0