How to efficiently merge two dataframe on two columns

Asked Jun 26 '18 at 15:54

Active Jun 22 '19 at 04:04

Viewed 42 times

I have two dataframes which are quite big (2 000 000 rows approximately)

They both have two columns in common ('CD' and 'BA') and I would like to join my dataframes on those two columns.

I have many solution, but for now they are all taking way to long (more than 7 secs)

affect = df1.merge(df2, on=['BA', 'CD'], how='left')

affect = df1.set_index(['BA', 'CD']).join(df2.set_index(['BA', 'CD']), how='left')

df1.set_index(['BA', 'CD'], inplace=True)
df2.set_index(['BA', 'CD'], inplace=True)
affect = df1.join(df2, how='left')

Do you have any idea how to speed things up ?

edited Jun 22 '19 at 04:04

cs95

379,657
97
704
746

asked Jun 26 '18 at 15:54

Louis

2

https://stackoverflow.com/questions/50886269/efficient-way-to-merge-multiple-large-dataframes/50923275#50923275 might help – cs95 Jun 26 '18 at 15:56
1

And 7 secs for a 2 Million rows data is not bad too. – Bharath M Shetty Jun 26 '18 at 15:57
1

@coldspeed - It is left join, so `concat` is not possible use - it is used only for inner and outer join. – jezrael Jun 26 '18 at 16:00
@jezrael oh right, thank you – cs95 Jun 26 '18 at 16:00

How to efficiently merge two dataframe on two columns

0 Answers0