How to sort rows of dataframe in pyspark

Question

I have a dataframe with two columns which contain numbers and i need to sort the dataframe row wise and not column wise. Everywhere it is given how to sort a dataframe by column but nowhere I can find how to sort all the rows of dataframe in pyspark

col1    col2

2        1

3        2

Expected output

col1   col2

1       2

2       3

You may need to provide sample data along with the desired result. This [link](https://stackoverflow.com/help/mcve) will help you update/ refine your question. — Prem, Nov 14 '17 at 05:37

score 1 · Answer 1 · answered Nov 14 '17 at 07:46

You may need some workaround to produce your desired result.

Here is an example to sort data based on a row.

From your dataframe, you may need create an index first.

df = spark.createDataFrame([['index1',3,2,1], ['index2',2,1,3]], ['index', 'a', 'b', 'c']) 
columns = [i for i in df.columns if i != 'index'] 
df.show()

def sort_row_df(row_to_sort):
    row_data = df.filter(col('index')==row_to_sort).collect()[0] 

    sorted_row = sorted([[row_data[col_], col_] for col_ in columns])

    rearrange_col = [i[1] for i in sorted_row]

    return df.select("index", *rearrange_col)

Lets say you wish to sort based on row 'index1',

row_to_sort = 'index1'
sorted_df = sort_row_df(row_to_sort)
sorted_df.show()

To sort based on row 'index2',

row_to_sort = 'index2'
sorted_df = sort_row_df(row_to_sort)
sorted_df.show()

If you want to sort all data based on rows, i would suggest you just to transpose all the data, sorts it, and transpose it back again. You may refer on how to transpose df in pyspark.

How to sort rows of dataframe in pyspark

1 Answers1