0

I get exception when trying to read dataset from s3. Company case class contains set with Employee case classe. 

Exception in thread "main" java.lang.UnsupportedOperationException: No Encoder found for Set[com.model.company.common.Employee]
- field (class: "scala.collection.immutable.Set", name: "employees")
- field (class: "com.model.company.Company", name: "company")

I tried with kryo:

implicit def myDataEncoder[T]: Encoder[Set[Employee]] = Encoders.kryo[scala.collection.immutable.Set[Employee]

but it also doesn't work. Do you know how to convert scala collection to dataset ?

Code:

val sqlContext = sparkSession.sqlContext
import sqlContext.implicits._

val records = sparkSession.read.json(s"s3a://${config.input.fullPath}").as[Company]
Przemek
  • 208
  • 2
  • 8

1 Answers1

0

Change from List,Set -> Seq

More information about dataset data type you can find here: https://spark.apache.org/docs/latest/sql-programming-guide.html#data-types

Przemek
  • 208
  • 2
  • 8