Python Pandas: get rows of a DataFrame where a column is not null

Question

I'm filtering my DataFrame dropping those rows in which the cell value of a specific column is None.

df = df[df['my_col'].isnull() == False]

Works fine, but PyCharm tells me:

PEP8: comparison to False should be 'if cond is False:' or 'if not cond:'

But I wonder how I should apply this to my use-case? Using 'not ...' or ' is False' did not work. My current solution is:

df = df[df['my_col'].notnull()]

@MohammadAthar that doesn't work. As mentioned in my question. — Matthias, Apr 05 '18 at 13:19
@Matthias, i agree with PEP8 - it doesn't look Pythonic or Pandaic (Pandas has boolean indexing and negation operator `~`)... — MaxU - stand with Ukraine, Apr 05 '18 at 13:19
PEP8 cannot apply here. You are comparing a Series against a single value. You don't have a scalar. If you want to get rid of it, you can just use `df['my_col'].notnull()` though. — ayhan, Apr 05 '18 at 13:22
@jezrael that is definitely not the same question. This should not have been closed for that reason. — FHTMitchell, Apr 05 '18 at 13:22

FHTMitchell · Accepted Answer · 2018-04-05T13:34:29.403

So python has the short-circuiting logic operators not, and, or. These have a very specific meaning in python and cannot be overridden (not must return a bool and a and/or b always returns either a or b or throws an error.

However, python also has over-loadable boolean operators ~ (not), & (and), | (or) and ^ (xor).

You may recognise these as the int bitwise operators, but Numpy (and therefore pandas) use these to do array / series boolean operations.

For example

b = np.array([True, False, True]) & np.array([True, False, False])
# b --> [True False False]
b = ~b 
# b --> [False True True]

Hence what you want is

df = df[~df['my_col'].isnull()]

I agree with PEP8, don't do == False.

Thanks for the explantation and yet another example. – Matthias Apr 05 '18 at 14:50 — Matthias, Apr 05 '18 at 14:50

Python Pandas: get rows of a DataFrame where a column is not null

1 Answers1

Linked