Spark : Is there an equivalent to spark SQL's LATERAL VIEW in the Spark API?

Question

Title says it all:

Is there an equivalent to the SPARK SQL LATERAL VIEW command in the Spark API so that I can generate a column from a UDF that contains a struct of multiple columns worth of data, and then laterally spread the columns in the struct into the parent dataFrame as individual columns?

Something equivalent to df.select(expr("LATERAL VIEW udf(col1,col2...coln)"))

Hey thanks for answering. I actually solved this by selecting them by their tuple handles... see my answer below... — Rimer, Feb 26 '21 at 17:27
I think explode only blows out ARRAYS, not struct of a single set of values... I might be wrong? — Rimer, Feb 26 '21 at 17:32

score 0 · Accepted Answer · answered Feb 26 '21 at 17:30

I solved this by selecting the udf into a column:

val dfWithUdfResolved = dataFrame.select(calledUdf()).as("tuple_column"))

... then ...

dfWithUdfResolved
  .withColumn("newCol1", $"tuple_column._1")
  .withColumn("newCol2", $"tuple_column._2")
  // ...
  .withColumn("newColn", $"tuple_column._n")

Basically using tuple notation to pull values out of the column into new discrete columns.

Spark : Is there an equivalent to spark SQL's LATERAL VIEW in the Spark API?

1 Answers1