Following is my dataframe:
Name Len
Tom 3
Tommy 5
Rock 4
Rocky 5
I need following output dataframe:
Name Len New_Name New_Len
Tom 3 Tommy 5
Tommy 5 Rock 4
Rock 4 Rocky 5
Rocky 5 ... ...
With Pandas I can achieve this using
df['new_name'] = df['name'].shift(-1)
df['new_len'] = df['len'].shift(-1)
Can I get any suggestions on how to achieve the same in PySpark? Is there any equivalent of pandas.dataframe.shift()
method in PySpark?