0

I want to implement a generic encoder because I have a small project on Spark 1.6 and when I migrate it to the spark 2.0 its giving me the warning and error of 

Unable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._  Support for serializing other types will be added in future releases.

at many places hence I want to implement a generic encoder and put it in package object. I just want to know how can I implement a generic Encoder in spark 2.0 ?

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
Shivansh
  • 3,454
  • 23
  • 46
  • It's just a trait http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Encoder – Alec Aug 09 '16 at 05:30

1 Answers1

2

Developing custom encoders usually starts with composing existing encoders from org.apache.spark.sql.Encoders object.

Please note that many places in Spark check whether the encoder in use is an instance of ExpressionEncoder (i.e. SparkSession.createDataset) so you may want to review expression encoders as the base for custom development.

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420