I have a List of dataframes: val dfs= Seq[df1,df2,df3,df4,....]
df1:
+--------+--------+
| value1 | Amount |
+--------+--------+
| A | 300 |
+--------+--------+
| B | 400 |
+--------+--------+
| C | 200 |
+--------+--------+
df2
+--------+--------+
| Value1 | Amount |
+--------+--------+
| A1 | 2000 |
+--------+--------+
| B1 | 100 |
+--------+--------+
| C1 | 349 |
+--------+--------+
df3:
+--------+--------+
| value1 | Amount |
+--------+--------+
| A2 | 20 |
+--------+--------+
| B2 | 34 |
+--------+--------+
| C2 | 56 |
+--------+--------+
df4:
+--------+--------+--------+
| value1 | value2 | amount |
+--------+--------+--------+
| A3 | CA | 234 |
+--------+--------+--------+
| B3 | AZ | 324 |
+--------+--------+--------+
| C3 | NY | 213 |
+--------+--------+--------+
I want to union all the dataframes in list, but due to difference in number of columns, unable to do it. For unionAll: when I do unionall dfs.reduce(unionall).show()
Instead of unionall first three dataframes , adding fourth column as null and perform the unionall with df4. Can someone help me how to do , unionall in generic way., taking the dataframes from list
how to do unionAll operation in loop condition for list of dataframes with different columns?
Expected output:
+--------+--------+--------+
| value1 | value2 | amount |
+--------+--------+--------+
| A | null | 300 |
+--------+--------+--------+
| B | null | 400 |
+--------+--------+--------+
| C | null | 200 |
+--------+--------+--------+
| A1 | null | 2000 |
+--------+--------+--------+
| B1 | null | 100 |
+--------+--------+--------+
| C1 | null | 349 |
+--------+--------+--------+
| A2 | null | 20 |
+--------+--------+--------+
| B2 | null | 34 |
+--------+--------+--------+
| C2 | null | 56 |
+--------+--------+--------+
| A3 | CA | 234 |
+--------+--------+--------+
| B3 | AZ | 324 |
+--------+--------+--------+
| C3 | NY | 213 |
+--------+--------+--------+
