3

Following is my dataframe:

Name    Len
Tom     3
Tommy   5
Rock    4
Rocky   5

I need following output dataframe:

Name    Len     New_Name    New_Len
Tom     3       Tommy       5
Tommy   5       Rock        4
Rock    4       Rocky       5
Rocky   5       ...         ...

With Pandas I can achieve this using

df['new_name'] = df['name'].shift(-1)
df['new_len'] = df['len'].shift(-1)

Can I get any suggestions on how to achieve the same in PySpark? Is there any equivalent of pandas.dataframe.shift() method in PySpark?

Community
  • 1
  • 1
Avinash Clinton
  • 543
  • 1
  • 8
  • 19

0 Answers0