Transposing rows to columns in PySpark

Question

how can I transpose one spark DataFrame in such a way:

From:

Key	Value
Key1	Value1
Key2	Value2
Key3	Value3

TO:

Key1	Key2	Key3
Value1	Value2	Value3

Thanks!

Kindly provide a minimal reproducible [example](https://stackoverflow.com/help/minimal-reproducible-example) — Abhyuday Vaish, Mar 14 '22 at 16:45
Please provide enough code so others can better understand or reproduce the problem. — Community, Mar 14 '22 at 16:45

score 4 · Accepted Answer · answered Mar 14 '22 at 16:58

You can apply pivot operation to transpose rows to columns.


from pyspark.sql import functions as F

data = [("Key1", "Value1", ),
("Key2", "Value2", ),
("Key3", "Value3", ), ]

df = spark.createDataFrame(data, ("Key", "Value", ))

df.groupBy().pivot("Key").agg(F.first("Value")).show()

"""
+------+------+------+
|  Key1|  Key2|  Key3|
+------+------+------+
|Value1|Value2|Value3|
+------+------+------+
"""

score 1 · Answer 2 · answered Mar 14 '22 at 16:58

pyspark.sql.GroupedData.pivot

df = spark.createDataFrame([('key1','value1'),('key2','value2'),('key3','value3')], ['key', 'value'])

import pyspark.sql.functions as F

df.groupBy().pivot('key').agg(F.first('value')).show()

or

df.groupBy().pivot('key').agg({"value":"first"}).show()

+------+------+------+
|  key1|  key2|  key3|
+------+------+------+
|value1|value2|value3|
+------+------+------+

Transposing rows to columns in PySpark

2 Answers2