Looping through dataframe list with IF Statement

Question

Just running a simple for-loop on a list of dataframes, however trying to add an IF clause... and it keeps erroring out.

df_list = [df1, df2, df3]
for df in df_list:
   if df in [df1, df2]:
      x = 1
   else:
      x = 2
.
.
.
ValueError: Can only compare identically-labeled DataFrame objects

Above is a simplified version of what I'm attempting. Can anyone tell me why this isn't working and a fix?

the error message is self-explanatory. Maybe try it with equality? — Srinivas Reddy Thatiparthy, Mar 31 '22 at 15:42
I smell an XY problem here. `if` statements in `for` loops are one thing, but your problem seems to come in when trying to evaluate if one dataframe is one of the dataframes in a list. Do you want to know if it is the same object? Or just has the same index, column, and values? Your question is ambiguous. — piRSquared, Mar 31 '22 at 16:17
@chicagobeast12 Then you should take my answer. `.equals()` adds burdens to comparing the values, but what you just need is to compare the identity. — Panwen Wang, Mar 31 '22 at 19:28

score 3 · Answer 1 · answered Mar 31 '22 at 15:43

3

You could use DataFrame.equals with any instead:

df_list = [df1, df2, df3]
for df in df_list:
    if any(df.equals(y) for y in [df1, df2]):
        x = 1
    else:
        x = 2

answered Mar 31 '22 at 15:43

Panwen Wang · Accepted Answer · 2022-03-31T16:48:50.920

2

Do NOT use .equals() here!

It's unnecessary and slowing down you program, use id() instead:

df_list = [df1, df2, df3]
for df in df_list:
   if id(df) in [id(df1), id(df2)]: 
      x = 1
   else:
      x = 2

Because here you just need to compare the identities, rather than the values.

edited Mar 31 '22 at 16:48

answered Mar 31 '22 at 16:38

Panwen Wang

3,573
1
18
39

Depending upon the use case `id` can be nice, but it could also be unwanted. For instance if `df3` was created with `df3=df1`, then they share the same `id`, yet for _some reason_ perhaps they should be handled differently. Guess that could be avoided with `df3=df1.copy()` so it's truly a different object, not just a reference – ALollz Apr 01 '22 at 18:12
If `df3=df1` is in the case, then @ALollz 's answer will be your choice. Neither `id()` nor `.equals()` can distinguish them. But `id()` is able to tell `df3` from `df1` if `df3` is a copy of `df1`, while `.equals()` is not. – Panwen Wang Apr 01 '22 at 19:07

ALollz · Answer 3 · 2022-03-31T15:52:59.057

1

You could use a better container and reference them by labels.

Equality checks for large DataFrames with object types can become slow, >> seconds, but it will take ~ns to check if the label is in a list.

dfs = {'df1': df1, 'df2': df2, 'df3': df3}
for label, df in dfs.items():
    if label in ['df1', 'df2']:
        x = 1
    else:
        x = 2

edited Mar 31 '22 at 15:52

answered Mar 31 '22 at 15:47

ALollz

57,915
7
66
89

score 0 · Answer 4 · answered Mar 31 '22 at 15:46

0

You need to use df.equals()

https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.equals.html

df_list = [df1, df2, df3]
for df in df_list:
   if df.equals(df1) or df.equals(df2):
      # blah blah

answered Mar 31 '22 at 15:46

mortice

84
4

score 0 · Answer 5 · answered Mar 31 '22 at 15:54

The following link might help: Pandas "Can only compare identically-labeled DataFrame objects" error

According to this, the data frames being compared with == should have the same columns and index otherwise it gives the error.

Alternatively, you can compare the data frames using dataframe.equals method. Please refer to the documentation below: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.equals.html

Looping through dataframe list with IF Statement

5 Answers5