1

This is different from How to create Spark Dataset or Dataframe from case classes that contains Enums. In this question I want to know how to create a Dataframe, not a DataSet.

I have been trying to create Spark Dataframe using case classes that contain Enums but I'm not able to. I'm using Spark version 1.6.0. The exceptions is complaining about that the schema is not supported for my Enum. Is this not possible in Spark, to have enums in the data and create dataframes?

Code:

import org.apache.spark.sql.SQLContext
import org.apache.spark.{SparkConf, SparkContext}

object MyEnum extends Enumeration {
  type MyEnum = Value
  val Hello, World = Value
}

case class MyData(field: String, other: MyEnum.Value)

object EnumTest {

  def main(args: Array[String]): Unit = {
    val sparkConf = new SparkConf().setAppName("test").setMaster("local[*]")
    val sc = new SparkContext(sparkConf)
    val sqlCtx = new SQLContext(sc)

    import sqlCtx.implicits._

    val df = sc.parallelize(Array(MyData("hello", MyEnum.World))).toDF()

    println(s"df: ${df.collect().mkString(",")}}")
  }

}

Exception:

Exception in thread "main" java.lang.UnsupportedOperationException: Schema for type com.nordea.gpdw.dq.MyEnum.Value is not supported

Community
  • 1
  • 1
ErikHabanero
  • 355
  • 3
  • 9
  • @zero323, can you please unmark it as a duplicate? – ErikHabanero Sep 30 '16 at 14:47
  • You won't get a better answer. If you're looking for 1.6 specific https://stackoverflow.com/questions/32440461/how-to-define-schema-for-custom-type-in-spark-sql/32454596#32454596 but it is not portable to 2.0. – zero323 Sep 30 '16 at 14:56
  • And please don't get me wrong but I don't think this should be reopened. You asked *How to create Spark Dataset or __Dataframe__* there using almost identical code and accepted and answer which satisfy you. There is no point in duplicating this here so I wont handle this myself. If you don't agree with that please ask for a moderator help (you can use flag link below the question). – zero323 Sep 30 '16 at 15:09

0 Answers0