3

I can read in a Spark dataframe as a custom object like this:

spark.read.csv("path/to/file").as[Gizmo]

But how can I convert a single Spark Row object to its equivalent case class? (If you're worried about why I want to do this, please consult this question.) Clearly Spark knows how to do this, but I don't see any straightforward way of accomplishing it (short of converting the Row into an RDD of length 1 and then converting back).

row.as[Gizmo] // doesn't work. What goes here?
Sasgorilla
  • 2,403
  • 2
  • 29
  • 56
  • https://stackoverflow.com/questions/28166555/how-to-convert-row-of-a-scala-dataframe-into-case-class-most-efficiently?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa – vvg May 22 '18 at 15:44
  • 2
    Did you try `row.asInstanceOf[Gizmo]` ? – philantrovert May 22 '18 at 17:03
  • 1
    @philantrovert - I get `java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast to Gizmo`. – Sasgorilla May 23 '18 at 11:08
  • 1
    Then you can pattern match: `val x = a match { case Row(a: String, b: Int) => Gizmo(a,b)}` – philantrovert May 23 '18 at 12:20

0 Answers0