16

I want to get all values of a column in pyspark dataframe. I did some search, but I never find a efficient and short solution.

Assuming I want to get a values in the column called "name". I have a solution:

sum(dataframe.select("name").toPandas().values.tolist(),[])

It works, but it is not efficient since it converts to pandas then flatten the list... Is there a better and short solution?

chen
  • 341
  • 1
  • 3
  • 11
  • 2
    There's no fast and efficient way to do it, but you *can* do way better than [using `sum` to flatten a list of lists](https://stackoverflow.com/a/41772165/5858851). The best you're going to get is probably: `[x["name"] for x in dataframe.select("name").collect()]` – pault Sep 05 '19 at 17:11

2 Answers2

27

Below Options will give better performance than sum.

Using collect_list

import pyspark.sql.functions as f
my_list = df.select(f.collect_list('name')).first()[0]

Using RDD: 

my_list = df.select("name").rdd.flatMap(lambda x: x).collect()

I am not certain but in my couple of stress test, collect_list gives better performance. Will be great if someone can confirm.

SMaZ
  • 2,515
  • 1
  • 12
  • 26
0

To get all values of a column in a list we can use collect() -

column_value_list = [row['<column_name>'] for row in df.select('<column_name>').collect()]

kartik
  • 79
  • 3