spark create dataset from typed sequence

Question

Similar to the spark documentation here:

http://spark.apache.org/docs/latest/sql-programming-guide.html
case class Person(name: String, age: Long)
val caseClassDS = Seq(Person("Andy", 32)).toDS()
caseClassDS.show()

error for a sequence of Seq[org.opengis.feature.simple.SimpleFeature]:

/geomesaSparkFirstSteps/src/main/scala/myOrg/GeoInMemory.scala:162: value toDS is not a member of Seq[org.opengis.feature.simple.SimpleFeature]
[error]   geoResult.toDS()

see https://github.com/geoHeil/geomesaSparkFirstSteps/blob/master/src/main/scala/myOrg/GeoInMemory.scala#L162 for details

How can I fix this statement? Is the encoder from Seq[someObject] issing?

@Downvoter please explain why. – Georg Heiler Mar 06 '17 at 16:26 — Georg Heiler, Mar 06 '17 at 16:26

score 2 · Accepted Answer · edited May 23 '17 at 12:32

Conversions between Seq and Dataset require an implicit Encoder for the stored class.

implicit def localSeqToDatasetHolder[T](s: Seq[T])(
  implicit arg0: Encoder[T]): DatasetHolder[T]

Product types (like cases classes) containing common Scala types use implicit Encoders provided by SparkSession.implicits. For arbitrary class you'll have to use generic Java or Kryo encoder. See How to store custom objects in Dataset? for details.

spark create dataset from typed sequence

1 Answers1