0

I have list of columns in a list need to add withcolumn like similary how we do it in scala like below:

list.foldLeft(df){(tempDF, listValue) => tempDF.withColumn(listValue._1, listValue._2) }

Let me clarify the question. I have a dataframe with struct of array types I need to flatten each and every array using explode eg: df.withcolumn(col1,explode(col1)) if need to create a generic function to pass all column in one go how do u do it? if I use for loop it is doubling the data for every explode statement

How can you do it in pyspark?

Nikunj Kakadiya
  • 2,689
  • 2
  • 20
  • 35
Matrix
  • 9
  • 1
  • 4
  • 1
    Does this answer your question? [What is the 'pythonic' equivalent to the 'fold' function from functional programming?](https://stackoverflow.com/questions/10366374/what-is-the-pythonic-equivalent-to-the-fold-function-from-functional-program) – Lamanus Mar 04 '20 at 15:28
  • Since I found this answer, I have deleted mine. – Lamanus Mar 04 '20 at 15:29
  • Let clarify the question i have a dataframe with struct of array types i need to flatten each and every array using explode eg: df.withcolumn(col1,explode(col1)) if need to create a generic function to pass all column in one go how do u do it? if i use for loop it is doubling the data for every explode statement – Matrix Mar 04 '20 at 15:37
  • Difficult to help you without a minimal reproducible example (complete with expected output). Without knowing more, you might consider joining all of your arrays together before performing an explode. – CPak Mar 04 '20 at 17:09

0 Answers0