Python/Pandas, .count not working on larger data frame

Question

I am trying to count the number of True/False values in a data frame like this:

df = pd.DataFrame({'a': [True, False, True],
                  'b': [True, True, True],
                  'c': [False, False, True]})
count_cols = ['a', 'b', 'c']
df['count'] = df[df[count_cols] == True].count(axis=1)

This is working fine on this example. But when I test it on my actual df (shape - (25168, 303)), I am getting the following error:

I Understood from - What does `ValueError: cannot reindex from a duplicate axis` mean? - that this usually occurs when there are duplicate values in the index and I have tried both df.reindex() and df[~df.index.duplicated()], but I am still getting the same error message.

have you tried `df.sum(axis=1)` ? – Andy L. Sep 09 '19 at 10:07 — Andy L., Sep 09 '19 at 10:07
Thanks, but that throws the same error – Maverick Sep 09 '19 at 10:57 — Maverick, Sep 09 '19 at 10:57

jezrael · Accepted Answer · 2019-09-09T11:00:38.690

1

Filter columns by list and count Trues values by sum - Trues are processing like 1s:

df['count'] = df[count_cols].sum(axis=1)
print (df)
       a     b      c  count
0   True  True  False      2
1  False  True  False      1
2   True  True   True      3

EDIT: For avoid error one possible solution is convert values to numpy array:

df['count'] = np.sum(df[count_cols].values, axis=1)
print (df)
       a     b      c  count
0   True  True  False      2
1  False  True  False      1
2   True  True   True      3

edited Sep 09 '19 at 11:00

answered Sep 09 '19 at 10:13

jezrael

822,522
95
1,334
1,252

Thanks, but `df[count_cols].sum(axis=1)` throws the same error – Maverick Sep 09 '19 at 10:58
Perfect, thank you! Do you have any idea why mine works on the test df but not the big one? – Maverick Sep 09 '19 at 14:50
1

@Maverick It seems some data related issue, hard to know – jezrael Sep 09 '19 at 15:54

Python/Pandas, .count not working on larger data frame

1 Answers1