Iterate a spark dataframe with static list of values using withcolumn

Question

I am bit new on pyspark. I have a spark dataframe with about 5 columns and 5 records. I have list of 5 records. Now I want to add these 5 static records from the list to the existing dataframe using withColumn. I did that, but its not working. Any suggestions are greatly appreciated.

Below is my sample:

dq_results=[] 

for a in range(0,len(dq_results)):
    dataFile_df=dataFile_df.withColumn("dq_results",lit(dq_results[a]))
    print lit(dq_results[a])

thanks, Sreeram

Thanks for your response..but my question is adding a column with custom list of values. The link provided, adds the new column from the existing dataframe. — sreeram ch, Feb 01 '18 at 15:04

score 1 · Accepted Answer · edited Feb 01 '18 at 20:44

1

dq_results=[]

Create one data frame from list dq_results:

df_list=spark.createDataFrame(dq_results_list,schema=dq_results_col)

Add one column for df_list id (it will be row id)

df_list_id = df_list.withColumn("id", monotonically_increasing_id())

Add one column for dataFile_df id (it will be row id)

dataFile_df= df_list.withColumn("id", monotonically_increasing_id())

Now we can join the both dataframe df_list and dataFile_df.

dataFile_df.join(df_list,"id").show()

So dataFile_df is final data frame

edited Feb 01 '18 at 20:44

pault

41,343
15
107
149

answered Jan 31 '18 at 11:03

Rahul Kumar Singh

117
1
14

Hope this will solve your problem – Rahul Kumar Singh Jan 31 '18 at 11:04
Thanks so much it worked.. – sreeram ch Feb 01 '18 at 17:17
@sreeramch welcome for pysaprk- hadoop let me know for any help – Rahul Kumar Singh Feb 02 '18 at 12:35
This solution might not work, as monotonically_increasing_id() varies with the partitioning of data and is non deterministic. You can zip with an index instead, that is, in contrast to the suggestions, deterministic. – gilgamash May 02 '19 at 09:07

score 0 · Answer 2 · answered Jan 31 '18 at 04:56

withColumn will add a new Column, but I guess you might want to append Rows instead. Try this:

df1 = spark.createDataFrame([(a, a*2, a+3, a+4, a+5) for a in range(5)], "A B C D E".split(' '))

new_data = [[100 + i*j for i in range(5)] for j in range(5)]

df1.unionAll(spark.createDataFrame(new_data)).show()

+---+---+---+---+---+
|  A|  B|  C|  D|  E|
+---+---+---+---+---+
|  0|  0|  3|  4|  5|
|  1|  2|  4|  5|  6|
|  2|  4|  5|  6|  7|
|  3|  6|  6|  7|  8|
|  4|  8|  7|  8|  9|
|100|100|100|100|100|
|100|101|102|103|104|
|100|102|104|106|108|
|100|103|106|109|112|
|100|104|108|112|116|
+---+---+---+---+---+

Thanks for your response..but i need to add a new column with dynamic values from the list I have it to the dataframe.. — sreeram ch, Feb 01 '18 at 13:42

Iterate a spark dataframe with static list of values using withcolumn

2 Answers2