How to select a random text value from a python list using pyspark?

Question

Is there a way to select a random text value from following python list using pyspark:-

data_list = ["abc", "xyz", "pqr"]

I know that I can implement a pyspark UDF which will return a random text value from python list using random.choice() python function but do we have an alternative function in pyspark itself to do the same?

The main reason to do the above is to create a huge dataset of 1000 rows by randomly selecting elements from the python data_list as mentioned above.

A random element of a python list, or a random element from a pyspark RDD or sql DataFrame? — Dave, May 13 '20 at 14:00

score 1 · Accepted Answer · answered May 13 '20 at 14:43

You can create dataframe from the list then using rand function we can generate random data from the list.

Example:

data_list = ["abc", "xyz", "pqr"]

from pyspark.sql.functions import *

spark.createDataFrame([(k,) for k in data_list]).orderBy(rand()).limit(1).show()
#+---+
#| _1|
#+---+
#|pqr|
#+---+

spark.createDataFrame([(k,) for k in data_list]).orderBy(rand()).limit(1).show()
#+---+
#| _1|
#+---+
#|xyz|
#+---+

#to get string
spark.createDataFrame([(k,) for k in data_list]).orderBy(rand()).limit(1).rdd.map(lambda x:x['_1']).collect()[0]
#u'abc'

How to select a random text value from a python list using pyspark?

1 Answers1

Linked