Creating Separate Spark dataframe from existing arraytype column

Question

I have a spark dataframe as

with schema

StructType(structField("a",IntegerType,False),structField("b",IntegerType,False),structField("c",ArrayType(structType(structField("d",IntegerType,False),structField("e",IntegerType,False)))

I want to create a separate dataframe from column "c" which is of array type.

Desired output format is

@GamingFelix Thanks for your help. Explode is not dynamically creating separate columns. Can you please help me with exact syntax you are referring to, where you can have n number of columns in array. — Ankush Monga, Aug 25 '20 at 15:53
Hi! does this help https://stackoverflow.com/questions/51609740/spark-dataframe-explode-list-column ? — GamingFelix, Aug 25 '20 at 19:04

score 1 · Answer 1 · answered Aug 25 '20 at 16:50

1

Try this-

df.selectExpr("a", "b", "inline_outer(c)").show()

answered Aug 25 '20 at 16:50

Som

6,193
1
11
22

Wow. this is really useful. – Lamanus Aug 26 '20 at 02:42

Creating Separate Spark dataframe from existing arraytype column

1 Answers1