How can dataframe with list of lists can be explode each line as columns - pyspark

Question

I have a data frame as below

--------------------+
|                pas1|
+--------------------+
|[[[[H, 5, 16, 201...|
|[, 1956-09-22, AD...|
|[, 1961-03-19, AD...|
|[, 1962-02-09, AD...|
+--------------------+

want to extract few columns from each row from above 4 rows and create a dataframe like below . Column names should be from the schema not hard coded ones like column1 & column2.

---------|-----------+
| gender | givenName |
+--------|-----------+
|      a |       b   |
|      a |       b   |
|      a |       b   |
|      a |       b   |
+--------------------+

pas1 - schema
root
|-- pas1: struct (nullable = true)
|    |-- contactList: struct (nullable = true)
|    |    |-- contact: array (nullable = true)
|    |    |    |-- element: struct (containsNull = true)
|    |    |    |    |-- contactTypeCode: string (nullable = true)
|    |    |    |    |-- contactMediumTypeCode: string (nullable = true)
|    |    |    |    |-- contactTypeID: string (nullable = true)
|    |    |    |    |-- lastUpdateTimestamp: string (nullable = true)
|    |    |    |    |-- contactInformation: string (nullable = true)
|    |-- dateOfBirth: string (nullable = true)
|    |-- farePassengerTypeCode: string (nullable = true)
|    |-- gender: string (nullable = true)
|    |-- givenName: string (nullable = true)
|    |-- groupDepositIndicator: string (nullable = true)
|    |-- infantIndicator: string (nullable = true)
|    |-- lastUpdateTimestamp: string (nullable = true)
|    |-- passengerFOPList: struct (nullable = true)
|    |    |-- passengerFOP: struct (nullable = true)
|    |    |    |-- fopID: string (nullable = true)
|    |    |    |-- lastUpdateTimestamp: string (nullable = true)
|    |    |    |-- fopFreeText: string (nullable = true)
|    |    |    |-- fopSupplementaryInfoList: struct (nullable = true)
|    |    |    |    |-- fopSupplementaryInfo: array (nullable = true)
|    |    |    |    |    |-- element: struct (containsNull = true)
|    |    |    |    |    |    |-- type: string (nullable = true)
|    |    |    |    |    |    |-- value: string (nullable = true)

Thanks for the help

Hi @darla, welcome to Stack Overflow! I ask you to elaborate a little bit your question. What do you mean for "not hardcoded"? You maybe want to flatten the dataframe? — Andrea, Oct 18 '18 at 09:04

score 0 · Accepted Answer · answered Oct 18 '18 at 09:10

If you want to extract few columns from a dataframe containing structs, you can simply do something like this:

from pyspark.sql import SparkSession,Row
spark = SparkSession.builder.appName('Test').getOrCreate()
df = spark.sparkContext.parallelize([Row(pas1=Row(gender='a', givenName='b'))]).toDF()

df.select('pas1.gender','pas1.givenName').show()

Instead, if you want to flatten your dataframe, this question should help you: How to unwrap nested Struct column into multiple columns?

How can dataframe with list of lists can be explode each line as columns - pyspark

1 Answers1