How to convert an array of dataFrames into a single data frame?

Question

I have an array of data frames called "dataFrames" and looks like this:

dataFrames(0)
+----------+--------------------+---------+-------------+
|Periodo   |              frutas|freq     |prods_qty    |
+----------+--------------------+---------+-------------+
|         1|Apple, Watermelon   |        1|            2|
|         1|Banana, StrawBerry  |        2|            2|
+----------+--------------------+---------+-------------+

dataFrames(1)
+----------+--------------------+---------+-------------+
|Periodo   |              frutas|freq     |prods_qty    |
+----------+--------------------+---------+-------------+
|         2|Naranjas, Fresas    |        7|            2|
|         2|Pineapple, Apples   |        9|            2|
+----------+--------------------+---------+-------------+

Well, I need to get a single dataframe like this:

+----------+--------------------+---------+-------------+
|Periodo   |              frutas|freq     |prods_qty    |
+----------+--------------------+---------+-------------+
|         1|Apple, Watermelon   |        1|            2|
|         1|Banana, StrawBerry  |        2|            2|
|         2|Naranjas, Fresas    |        7|            2|
|         2|Pineapple, Apples   |        9|            2|
+----------+--------------------+---------+-------------+

For this example the length of the array is 1, but the array could any size.

It is possible to achive this... or i need to store the dataframes into a hive table?

Thanks in advance

Thanks Ramesh, but as i was telling you... the array could have many time of periods, not just two. — LeeFernan, May 01 '18 at 07:02
what do you mean by many time of periods . If you can explains a little bit — Ramesh Maharjan, May 01 '18 at 07:05
DataFrames could have size of tree, in that case i need to convert: dataFrames(0) + dataFrames(1) + dataFrames(2) into a single DataFrame. Or could have size of 20... That makes sense? — LeeFernan, May 01 '18 at 07:11
so what you are saying is that you can have 20 dataframes like that and you want to merge them into one on one go. is that correct? — Ramesh Maharjan, May 01 '18 at 07:18

score 0 · Accepted Answer · answered May 01 '18 at 07:50

0

You can reduce a sequence or array of DataFrames together using unionAll:

val dfs = Array(df1, df2, df3)

val all = dfs.reduce(_ unionAll _)

answered May 01 '18 at 07:50

DNA

42,007
12
107
146

How to convert an array of dataFrames into a single data frame?

1 Answers1