Merging series of pandas dataframe into single dataframe

Question

I have a series of pandas data frame stored in variable df similar to below:

df

|  0    |   1    |
+-------+--------+
|ABCD   |   WXYZ |

|  0    |   1    |
+-------+--------+
|DEFJ   |   HJKL |

|  0    |   1    |
+-------+--------+
|ZXCT   |   WYOM |

|  0    |   1    |
+-------+--------+
|TYZX   |   NMEX |

I want to merge them to a single pandas data frame as below :

|  0    |   1    |
+-------+--------+
|ABCD   |   WXYZ |
|DEFJ   |   HJKL |
|ZXCT   |   WYOM |
|TYZX   |   NMEX |

So how can I merge series of pandas dataframe into one single pandas dataframe ?

please post the output of `df.head().to_dict()` so that we can reproduce your series of pandas data frame — Anurag Dabas, Apr 04 '21 at 07:11
@AnuragDabas: below is the output for ```df.head().to_dict()``` {0: {0: 'ABCD'}, 1: {0: WXYZ'}} {0: {0: 'DEFJ'}, 1: {0: 'HJKL'}} {0: {0: 'ZXCT'}, 1: {0: 'WYOM'}} {0: {0: 'TYZX'}, 1: {0: 'NMEX'}} — user3046211, Apr 04 '21 at 07:22

Laurent · Accepted Answer · 2021-04-04T08:40:40.847

1

As your code is now, you're only outputing one dataframe with one row only (overwriting the others).

Try this:

# Copy the names to pandas dataframes and save them in a list
import pandas as pd

dfs = []
for j in range(0,5):
  for i in divs[j].find_elements_by_tag_name('a'):
    i = i.get_attribute('text')
    i = parse_name(i)
    df = pd.DataFrame(i)
    df = df.transpose()
    dfs.append(df)

# Aggregate all dataframes in one
new_df = dfs[0]
for df in dfs[1:]:
    new_df = new_df.append(df)

# Update index
new_df = new_df.reset_index(drop=True)

# Print first five rows
new_df.head()
    0                   1
0   Lynn Batten         Emeritus Professor
1   Andrzej Goscinski   Emeritus Professor
2   Jemal Abawajy       Professor
3   Maia Angelova       Professor
4   Gleb Beliakov       Professor

edited Apr 04 '21 at 08:40

answered Apr 04 '21 at 07:21

Laurent

12,287
7
21
37

Unfortunately, it didn't work.It is not a list of data frames – user3046211 Apr 04 '21 at 07:45
What's the type of `df`, then? – Laurent Apr 04 '21 at 07:51
Ok, I understand your problem now, see my updated answer. – Laurent Apr 04 '21 at 08:41

score 0 · Answer 2 · answered Apr 04 '21 at 08:41

There are four ways to concat or merge dataframes, you may refer to this post

these are the most common implementations

import pandas as pd 

df1 = pd.DataFrame({0:['ABCD'], 1:['WXYX']})
df2 = pd.DataFrame({0:['DEFJ'], 1:['HJKL']})
df3 = pd.DataFrame({0:['ZXCT'], 1:['WYOM']})   
...
df = pd.concat([df1, df2, df3], axis=0)


print(df.head())

or if you have a list of dataframes with the same headers you can try

dfs = [df1, df2, df3 ..]
df = pd.concat(dfs, axis=0)

and the most simple way is to just use df.append

 df = df.append(anotherdf)

Merging series of pandas dataframe into single dataframe

2 Answers2

Linked