Confirming equality of two pandas dataframes?

Question

How to assert that the following two dataframes df1 and df2 are equal?

import pandas as pd
df1 = pd.DataFrame([1, 2, 3])
df2 = pd.DataFrame([1.0, 2, 3])

The output of df1.equals(df2) is False. As of now, I know two ways:

print (df1 == df2).all()[0]

or

df1 = df1.astype(float)
print df1.equals(df2)

It seems a little bit messy. Is there a better way to do this comparison?

@Divakar `np.allclose(df1, df2)` works for this case. But what if you have some strings in your dataframes as well? — Mehdi Jafarnia Jahromi, Jul 05 '16 at 21:10
@Divakar, could you please add it as an answer - it could help others in future? — MaxU - stand with Ukraine, Jul 05 '16 at 21:10
@MaxU Hmm I am not sure, was mostly a wild guess. Also, as OP pointed out for strings it might be producing unexpected output? — Divakar, Jul 05 '16 at 21:14
try this: `np.allclose(df1.select_dtypes(exclude=[object]), df2.select_dtypes(exclude=[object])) & df1.select_dtypes(include=[object]).equals(df2.select_dtypes(include=[object]))` - it's based on @Divakar's solution — MaxU - stand with Ukraine, Jul 05 '16 at 21:14
What do you mean by strings? Strings like `1.0` so that you want `'1.0' == 1` to return True? — ayhan, Jul 05 '16 at 21:16
@MaxU That looks like a reasonable solution! Seems to gracefully take care of strings. Consider posting that extended version as an answer. — Divakar, Jul 05 '16 at 21:19
@Divakar, the main idea belongs to you, i've just extended it a bit with a strings comparison, so please post it as an answer - it will definitely help people in future :) — MaxU - stand with Ukraine, Jul 05 '16 at 21:21
@MaxU I think you made the big changes needed there. My pandas knowledge is very limited, so I won't be able to put in words there. So, please do the honors! You can just attribute to the comments if needed :) — Divakar, Jul 05 '16 at 21:24

Alexander · Accepted Answer · 2018-08-30T20:40:19.163

40

You can use assert_frame_equal and not check the dtype of the columns.

# Pre v. 0.20.3
# from pandas.util.testing import assert_frame_equal

from pandas.testing import assert_frame_equal

assert_frame_equal(df1, df2, check_dtype=False)

edited Aug 30 '18 at 20:40

answered Jul 05 '16 at 22:40

Alexander

105,104
32
201
196

9

With pandas 0.20.3 `assert_frame_equal` is in the `pandas.testing` package: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.testing.assert_frame_equal.html – Matthew Turner Sep 08 '17 at 21:04
And important to notice, if there is no any output after the execution of assert_frame_equal function, then the two dataframes are equal. – Nabin Nov 01 '19 at 11:33

score 6 · Answer 2 · edited May 23 '17 at 10:28

Using elegant @Divakar's idea - numpy's allclose() will do the main trick for numbers:

In [128]: df1
Out[128]:
   0    s  n
0  1  aaa  1
1  2  aaa  2
2  3  aaa  3

In [129]: df2
Out[129]:
     0    s    n
0  1.0  aaa  1.0
1  2.0  aaa  2.0
2  3.0  aaa  3.0

In [130]: (np.allclose(df1.select_dtypes(exclude=[object]), df2.select_dtypes(exclude=[object]))
   .....:  &
   .....:  df1.select_dtypes(include=[object]).equals(df2.select_dtypes(include=[object]))
   .....: )
Out[130]: True

select_dtypes() will help you to separate strings and all other numeric dtypes

Confirming equality of two pandas dataframes?

2 Answers2

Linked