Can someone please give an example of how you would save a ML model in pySpark?
For
ml.classification.LogisticRegressionModel
I try to use the following:
model.save("path")
but it does not seem to work.
Asked
Active
Viewed 2.6k times
8
-
What is the error you encounter? Some more detail (your stack trace and code) might help – vkuo Apr 13 '16 at 23:07
-
1It just says that it could not find any method with the same name (I am using Spark 1.6.1 by the way) – ml_0x Apr 13 '16 at 23:08
-
Possible duplicate of [Save ML model for future usage](http://stackoverflow.com/questions/33027767/save-ml-model-for-future-usage) – Alberto Bonsanto Apr 14 '16 at 02:01
-
I'm running v2.2.0 and also get AttributeError: 'RandomForestRegressor' object has no attribute 'save'. google is failing me. all the docs seem to indicate this should work. – ericgtaylor Jan 23 '18 at 15:46
2 Answers
1
If I understand your question correctly, your method signature is incorrect.
According to the docs you also need to pass in your spark context.
vkuo
- 350
- 1
- 4
- 21
-
But this belongs to mllib. Does the ml inherit those functions? In any case, it sounds reasonable since I did not make use of the spark context (sc). Thanks! – ml_0x Apr 13 '16 at 23:14
-
This is better answered by http://stackoverflow.com/questions/30231840/difference-between-org-apache-spark-ml-classification-and-org-apache-spark-mllib however your methods should be the same. If this works, please accept this answer as the correct one! – vkuo Apr 13 '16 at 23:18
-
I will try it out tomorrow because I do not have access to pyspark right now. Thanks a lot! – ml_0x Apr 13 '16 at 23:22
0
In Spark 2.3.0, if you are using ML:
model.save("path")
Refer: Spark ML model .save
(I just ran LogisticRegression and saved it.)
But if you are using mllib, then as the other answer suggests use:
save(sc, path)
Refer: Spark MLLib model .save
Ani Menon
- 27,209
- 16
- 105
- 126