-2

I have a List list =["a","b","c"] and i have to add a new column to my dataframe, but first i have to construct it and this column have to be like:

x|y|z|list

strings in my list are columns, i mean that i have to construct my request like :

SELECT x,y,z, list FROM Dataframe

I tried to split strings in the list with

String.join("," , list)

but it is seen like a singl column not multiple columns

Dataset<Row> df= dataframe.withColumn("NewColumn", concat(dataframe.col("x"), lit("|"), dataframe.col("y"),lit("|"), String.join(","list));

Note 1: the size of my list is editable and the columns too Note 2: i have to call String.join(","list) in my function withColumn, i don't have the choice

the expected result is a dataframe :

 ------------------------------------------------------------
  x     y     z     a     b     c     **NewColumn**
 ------------------------------------------------------------
 val1  val2  val3  val4  val5  val6   val1|val2|val3|val4|val5|val6
 -------------------------------------------------------------

I don't see how to construct my new column, thank you for your help

yopi
  • 53
  • 1
  • 5

1 Answers1

0

Function concat will take params of type org.apache.spark.sql.Column but you are passing list of type String.

You have to convert list of String into list of Column type and pass those values to concat

def concat(exprs: org.apache.spark.sql.Column*): org.apache.spark.sql.Column

Below code is in scala, You can convert it into java.

val list = List("a","b","c")

dataframe.withColumn(
    "NewColumn", 
    concat(
        col("x"), 
        lit("|"), 
        col("y"),
        lit("|"),
        list.map(c => col(c)):_* // I have added this, You may need to convert your list of strings into list of columns, It will work.
    )
);
Srinivas
  • 8,957
  • 2
  • 12
  • 26
  • thank you but it doesn't answer my question... the issue is that i can't split the strings in the list and then i will construct my new column – yopi Nov 17 '20 at 15:17
  • my expected result is okay but my issue that when i tried to split my list into columns it is seen like a single column not multiple columns – yopi Nov 17 '20 at 15:32