5

I have a Datastructure that is recursive. Spark gives this error:

Exception in thread "main" java.lang.UnsupportedOperationException: cannot have circular references in class, but got the circular reference of class BulletPoint

As an example I did this code:

case class BulletPoint(item: String, children: List[BulletPoint])

object TestApp extends App {
  val sparkSession = SparkSession
    .builder()
    .appName("spark app")
    .master(s"local")
    .getOrCreate()

  import sparkSession.implicits._

  sparkSession.createDataset(List(BulletPoint("1", Nil), BulletPoint("2", Nil)))
}

Does someone have an idea how one can go around this issue?

10465355
  • 4,481
  • 2
  • 20
  • 44
oleber
  • 1,089
  • 4
  • 12
  • 25

1 Answers1

3

The exception is fairly explicit - such case is not supported by default. You have to remember that Datasets are encoded into relational schema, so all required fields have to be declared up-front and bounded. There is no place here for recursive structure.

There is small window here - binary Encoders:

import org.apache.spark.sql.{Encoder, Encoders}

sparkSession.createDataset(List(
  BulletPoint("1", Nil), BulletPoint("2", Nil)
))(Encoders.kryo[BulletPoint])

or equivalent:

implicit val bulletPointEncoder = Encoders.kryo[BulletPoint]

sparkSession.createDataset(List(
  BulletPoint("1", Nil), BulletPoint("2", Nil)
))

but it is really not something you'd like to have in your code, unless strictly necessary.

10465355
  • 4,481
  • 2
  • 20
  • 44
  • 1
    This seems to me the only solution, i was trying to see if there was some other solution. Big thanks for the support. – oleber Jan 17 '19 at 18:31