0

I have a spark Rdd which is of form Row(id,Words) Where words contains a list of words. I want to convert this list into a single column. Input

ID  Words
1   [w1,w2,w3]
2   [w3,w4]

I want to convert this to the output format

ID  Word
1   w1
1   w2
1   w3
2   w3
2   w4
vish
  • 67
  • 3
  • 10

1 Answers1

5

If you want to work rdd, you need to use flatMap():

rdd.flatMap(lambda x: [(x['ID'], w) for w in x["Words"]]).collect()
#[(1, u'w1'), (1, u'w2'), (1, u'w3'), (2, u'w3'), (2, u'w4')]

However, if you are open to using DataFrames (recommended) you can use pyspark.sql.functions.explode:

import pyspark.sql.functions as f
df = rdd.toDF()
df.select('ID', f.explode("Words").alias("Word")).show()
#+---+----+
#| ID|Word|
#+---+----+
#|  1|  w1|
#|  1|  w2|
#|  1|  w3|
#|  2|  w3|
#|  2|  w4|
#+---+----+

Or better yet, skip the rdd all together and create a DataFrame directly:

data = [
    (1, ['w1','w2','w3']),
    (2, ['w3','w4'])
]
df = sqlCtx.createDataFrame(data, ["ID", "Words"])
df.show()
#+---+------------+
#| ID|       Words|
#+---+------------+
#|  1|[w1, w2, w3]|
#|  2|    [w3, w4]|
#+---+------------+
pault
  • 41,343
  • 15
  • 107
  • 149
  • Flat map function creates multiple lists,I don't want the output to be in list format. Can I use explode function on two columns at once.I got another column which contains the scores of words.So I have to explode both the colums together – vish Feb 13 '18 at 23:09
  • @vish I don't understand. `flatMap()` should not create multiple lists. I think the best thing for you to do is to edit your question with your actual use case (or ask a new question). It's hard to give an answer without seeing the inputs and desired output _but_ you may be looking for `rdd.flatMap(lambda x: [(x['ID'], w, s) for w, s in zip(x["Words"], x["Scores"])]).collect()` – pault Feb 13 '18 at 23:13