3

I'm trying to write a custom Estimator to fit into Pipeline using Pyspark 1.6.0, so I try the solution in How to Roll a Custom Estimator in PySpark mllib

It works on Spark 2.0, but when I remove the TypeConverter parameter(which only exists in Spark 2.0) to test it Spark 1.6.0, it shows error:

ValueError: Param Param(parent='undefined', name='centered_threshold', doc='centered_threshold') does not belong to NormalDeviation_4dc98930f6cd0fd37ce6.

I'm not allowed to use Spark 2.0 in production. How can I change this code to make it work on Spark 1.6.0?

Community
  • 1
  • 1
Bamqf
  • 3,382
  • 8
  • 33
  • 47

0 Answers0