Apache Spark Scala with Play-json Validation

Question

java.lang.UnsupportedOperationException: Schema for type [trait object] is not supported

trait Container {
  def aa: String
  def bb: Int
}

case class First(aa: String, bb: Int) extends Container
case class Second(aa: String, bb: Int) extends Container

implicit val aaContainerFormat: Format[First] = Json.format[First]

implicit val bbContainerFormat: Format[Second] = Json.format[Second]

implicit def nodeContainerReads: Reads[Container] =
  try {
    Json.format[First].map(x => x: Container) or
    Json.format[Second].map(x => x: Container)
  } catch {
    case e: Exception => Reads {
      case _ => JsError(JsonValidationError("Cannot De-serialize value."))
    }
  }

implicit def nodeContainerWrites = new Writes[Container] {
  override def writes(node: Container): JsValue = node match {
    case a: First => Json.toJson(a)
    case b: Second => Json.toJson(b)
    case _ => Json.obj("error" -> "wrong Json")
  }
}

// Example Usage....
val spark: SparkSession = SparkSession.builder.appName("Unit Test").getOrCreate()
val js: Container = First("unit", "test")

spark.createDataFrame(Seq(js))

I expect the output of Datasets of [Container Object] but the actual output is java.lang.UnsupportedOperationException: Schema for type Container is not supported.

Cal · Answer 1 · 2019-01-24T03:35:21.607

0

Spark does not use typeclasses from Play JSON to convert Scala types into Spark SQL types. Instead you need to look at Spark Encoders which form the basis of conversions for Scala types to Spark types. If you have the Spark Session in scope, you can use import sparkSession.implicits._ so it will automatically create encoders for your case classes. I believe that Spark doesn't support sum-types out of the box so you would need to implement your own Encoder to somehow model that in Spark in an ad-hoc fashion. Please read here for more information if you want to encode sum types in Spark

edited Jan 24 '19 at 03:35

answered Jan 24 '19 at 03:24

Cal

734
1
9
25

I tried `import sparkSession.implicits._` but still the same error, I've tried implement custom Encoders but I still didn't find way yet. – Rex Jan 24 '19 at 06:30
So doing the implicits import will not help you if you want to represent a sum-type. You will need to take a look at https://stackoverflow.com/questions/36648128/how-to-store-custom-objects-in-dataset#answer-51957666 which goes into how you define a User Defined Type which is your use case – Cal Jan 24 '19 at 15:54

Apache Spark Scala with Play-json Validation

1 Answers1