Fetch values from nested struct in Spark DF

Asked Jun 14 '18 at 23:36

Active Jun 15 '18 at 00:08

Viewed 332 times

Spark DF with one column, where each row is of the

Type:

org.apache.spark.sql.Row

Form:

  col1: array (nullable = true)
|     |-- A1: struct (containsNull = true)
|     |       |-- B1: struct (nullable = true)
|     |       |    |-- B11: string (nullable = true)
|     |       |    |-- B12: string (nullable = true)

|     |       |-- B2: string (nullable = true)

I am trying to get the value of

A1->B1->B11.

Any methods to fetch this with the DataFrame APIs or indexing without converting each row into a seq and then iterating through it which affects my performance badly. Any suggestions would be great. Thanks

edited Jun 15 '18 at 00:08

Alper t. Turker

34,230
9
83
115

asked Jun 14 '18 at 23:36

data_person

4,194
7
40
75

4

Possible duplicate of [Querying Spark SQL DataFrame with complex types](https://stackoverflow.com/questions/28332494/querying-spark-sql-dataframe-with-complex-types) – Alper t. Turker Jun 15 '18 at 00:08
Do you mean: `df.select(explode($"col1")).select($"A1.B1.B11")` ? – Tom Lous Jun 15 '18 at 06:50
@TomLous Conceptually yes. – data_person Jun 15 '18 at 07:06
What about it doesn't -technically- work for you? – Tom Lous Jun 15 '18 at 08:31
1

@TomLous You mean try this exact command you have given? – data_person Jun 15 '18 at 08:34

Fetch values from nested struct in Spark DF

0 Answers0