0

I'm new to spark. I want to convert spark dataframe rows to case classes, but some of the fields are dynamic and the data type can't be decided beforehand. So I want to use some generic case classes.

case class Property[T: ClassTag](val value: T)

case class PropertyList(d: Map[String, Property[_]])

case class RowHolder(val id: Option[String] = None,
                     val properties: Option[PropertyList]= None)


udf {row => RowHolder(id = row.getAs[String]("id")}

It seems that Spark is having problem recognizing the generic type and convert it to schema, because I got the matcherror

Exception in thread "main" scala.MatchError: Property[_] (of class scala.reflect.internal.Types$ExistentialType)
        at org.apache.spark.sql.catalyst.ScalaReflection$class.getConstructorParameters(ScalaReflection.scala:838)

Is this related to spark type erasure? Will custom serialization be helpful? Any suggestions to resolve the issue?

Thanks!

cozyss
  • 1,290
  • 1
  • 15
  • 22
  • What if you cascade the `ClassTag` context bound throughout? `case class PropertyList[T: ClassTag](d: Map[String, Property[T]])` and `case class RowHolder[T: ClassTag](val id: Option[String] = None, val properties: Option[PropertyList[T]]= None)` – stefanobaghino Feb 23 '18 at 07:32
  • As a side note, constructor fields of `case class`es are implicitly `val`s, you don't have to specify it explicitly. Furthermore, it's a good practice to make your `case class`es `final`. You can read more about it here: https://stackoverflow.com/questions/34561614/should-i-use-the-final-modifier-when-declaring-case-classes – stefanobaghino Feb 23 '18 at 07:34
  • @stefanobaghino Thanks for the idea. I will take a look. The reason we can't use `ClassTag` is that we want to hold different data types in `PropertyList`, so it can have different values from the row. Otherwise, we need to create stringProperty, booleanProperty, etc. – cozyss Feb 23 '18 at 18:04
  • @stefanobaghino the custom serializer can't solve the problem right? – cozyss Feb 23 '18 at 18:17

0 Answers0