how to pass array of values to a new column of a dataframe

Asked Aug 22 '17 at 06:46

Active Aug 22 '17 at 06:51

Viewed 213 times

I have a data frame to which I want to add a column with the values which are present in an array. I want each value in the array to be placed in the column as each row. For Example:

val partitionName= "processingdatadate";
val partitionValue= "20170528,20170529,20170530";
var parts: Array[String] = partitionValue.split(",")
var partValues = parts.toSeq

I have a data frame 'loadParqDf' so to this I want to add a column with name 'processingdatadate'

loadParqDF = loadParqDF.withcolumn(partitionName, lit(partValues))

This method is not working because the output is not expected :

processingdatadate
----------------------------
[20170528,20170529,20170530]

I am expecting output as:

processingdatadate
------------------
20170528
20170529
20170530

edited Aug 22 '17 at 06:51

asked Aug 22 '17 at 06:46

Bhagya Sri

What is the output that you want? – philantrovert Aug 22 '17 at 06:48
hi philantrovert I have mentioned required output in my question. Please help me out. Any help would be appriciated. – Bhagya Sri Aug 22 '17 at 07:05
where are your other columns of your output dataframe? do they matter or not? – Ramesh Maharjan Aug 22 '17 at 07:25
@RameshMaharjan they dont matter . I just need the code which can fetch me the values in an array in each row. – Bhagya Sri Aug 22 '17 at 07:30
you can use `explode` in that case. you already have an array in that column. so using explode on that column would have each items in the array to be separate on each rows. :) – Ramesh Maharjan Aug 22 '17 at 07:34
@RameshMaharjan I dint understand your answer can u please elaborate it . Please? – Bhagya Sri Aug 22 '17 at 07:47
this question is already marked duplicate. didn't you find answer in the link ? – Ramesh Maharjan Aug 22 '17 at 07:52
try `loadParqDF.withcolumn(partitionName, explode(col(partitionName))).show`. for this you would need `import org.apache.spark.sql.functions._` – Ramesh Maharjan Aug 22 '17 at 08:34

how to pass array of values to a new column of a dataframe

0 Answers0