Exploding struct type large number of columns to two columns of each column having keys and values in Pyspark

Asked Jun 26 '20 at 12:57

Active Jun 26 '20 at 12:57

Viewed 67 times

I have a pyspark df who's schema looks like this

 root
 |-- company: struct (nullable = true)
 |    |-- A: long(nullable = true)
 |    |-- A_timestamp: long(nullable = true)
 |    |-- B: long(nullable = true)
 |    |-- B_timestamp: long(nullable = true)
 |    |-- C: long(nullable = true)
 |    |-- C_timestamp: long(nullable = true)
 |    |-- D: long(nullable = true)
 |    |-- D_timestamp: long(nullable = true)
 |    |-- E: long(nullable = true)
 |    |-- E_timestamp: long(nullable = true)

I want the final format of this dataframe with 4 columns to look like this

Please help me to solve this using Pyspark.

asked Jun 26 '20 at 12:57

Naresh Tambekar

a sample data will help here for more clear understanding – dsk Jun 26 '20 at 16:09
That's the sample data – Naresh Tambekar Jun 26 '20 at 18:18
can you also share how the input data looks like? – dsk Jun 26 '20 at 18:20
Is this the same question - https://stackoverflow.com/questions/62576297/type-casting-large-number-of-struct-fields-to-string-using-pyspark in a different format? did the answer to this previous question had any issue? – Raghu Jun 26 '20 at 23:26
Yeah It's the extension of the previous Question. This is more complex than I've expected – Naresh Tambekar Jun 29 '20 at 04:54

Exploding struct type large number of columns to two columns of each column having keys and values in Pyspark

0 Answers0