Loop over dataframes in pyspark

Question

I need help with this simple piece of code (pyspark):

def ann(table):
    table=table.withColumn('stand', lit('29Jan2020'))
for table in [akt_test, b60_test, db71_test, pek6_test, db00f_test, d23b_test, bw0110_test]:
    ann(table)

So I only try to add the column "stand" to all (already existing) dataframes from the list. Unfortunately the column is not added. Strangely, if i add a command "print(table.columns)" at the end of the function "ann", I see the new column there. But not in the actual data frames.

If i simply take one data frame and write

 akt_test=akt_test.withColumn('stand', lit('29Jan2020'))

everythings works fine. But not in a loop. I don't understand why and how I can heal it. Thanks in advance for Your ideas.

Answer 1

The function has to return the modified dataframe. In your function, only a copy of the table variable was modified, not the original dataframe variable.

Also, you need to apply the function to the list elements (e.g. using a list comprehension as below). If you put it in a for loop, the list elements won't be modified in place. Only the copies of the list elements will be modified, which are then discarded in the next iteration of the loop. See this question for example.

So, to make your code work, you can do this:

def ann(table):
    return table.withColumn('stand', lit('29Jan2020'))

df_list = [akt_test, b60_test, db71_test, pek6_test, db00f_test, d23b_test, bw0110_test]

df_list2 = [ann(df) for df in df_list]

If you want to keep the values of the original variables, you can do

akt_test, b60_test, db71_test, pek6_test, db00f_test, d23b_test, bw0110_test = [ann(df) for df in df_list]