Pyspark- equivalent of explode for splitting in to columns

Asked Apr 17 '18 at 14:22

Active Apr 17 '18 at 14:32

Viewed 123 times

I have a column in a dataframe that contains an array. I want to convert this array in to columns

>>> from pyspark.sql import Row
>>> from pyspark.mllib.linalg import DenseVector
>>> df = spark.createDataFrame([Row(a=1, intlist=DenseVector([1,2,3])), Row(a=2, intlist=DenseVector([4,5,6]))])
>>> df.show()
+---+-------------+
|  a|      intlist|
+---+-------------+
|  1|[1.0,2.0,3.0]|
|  2|[4.0,5.0,6.0]|
+---+-------------+

Expected output:

+---+---+---+---+
| a| _1| _2|  _3|
+---+---+---+---+
|  1|  1|  2|  3|
|  2|  4|  5|  6|
+---+---+---+---+

The explode function can do this but it adds rows instead of columns

>>> df.select(explode(df.intlist).alias("anInt")).show()
+-----+
|anInt|
+-----+
|    1|
|    2|
|    3|
|    4|
|    5|
|    6|
+-----+

Is there a way we can add columns instead of rows?

edited Apr 17 '18 at 14:28

asked Apr 17 '18 at 14:22

Clock Slave

7,627
15
68
109

How does column `a` turn into `_1`? And for your question, why not just use `df.select`? – ernest_k Apr 17 '18 at 14:26
@ErnestKiwele corrected the renaming. `df.select`? how, do you mean? – Clock Slave Apr 17 '18 at 14:29
@user6910411 is it advisable to go back an forth between an `rdd` and a `dataframe`? Performance wise. – Clock Slave Apr 17 '18 at 14:37
Both RDDs and UDFs are expensive here, but since you have `Vector` (not `array`), you won't be able to access this data otherwise in PySpark. – zero323 Apr 17 '18 at 14:44

Pyspark- equivalent of explode for splitting in to columns

0 Answers0