The difference between comparison to np.nan and isnull()

Question

I supposed that

data[data.agefm.isnull()]

and

data[data.agefm == numpy.nan]

are equivalent. But no, the first truly returns rows where agefm is NaN, but the second returns an empty DataFrame. I thank that omitted values are always equal to np.nan, but it seems wrong.

agefm column has float64 dtype:

(Pdb) data.agefm.describe()
count    2079.000000
mean       20.686388
std         5.002383
min        10.000000
25%        17.000000
50%        20.000000
75%        23.000000
max        46.000000
Name: agefm, dtype: float64

What does data[data.agefm == np.nan] mean exactly?

They are not the same: http://stackoverflow.com/questions/20320022/why-in-numpy-nan-nan-is-false-while-nan-in-nan-is-true — ayhan, Dec 27 '16 at 09:40
@Divakar So, if I should, is this a gap in *pandas* or I do a conceptual mistake? — sergzach, Dec 27 '16 at 09:44
Not really familiar with pandas isnull method, but we use np.isnan to detect NaNs with NumPy arrays. — Divakar, Dec 27 '16 at 09:45

piRSquared · Accepted Answer · 2016-12-27T10:11:48.137

38

np.nan is not comparable to np.nan... directly.

np.nan == np.nan

False

While

np.isnan(np.nan)

True

Could also do

pd.isnull(np.nan)

True

examples
Filters nothing because nothing is equal to np.nan

s = pd.Series([1., np.nan, 2.])
s[s != np.nan]

0    1.0
1    NaN
2    2.0
dtype: float64

Filters out the null

s = pd.Series([1., np.nan, 2.])
s[s.notnull()]

0    1.0
2    2.0
dtype: float64

Use odd comparison behavior to get what we want anyway. If np.nan != np.nan is True then

s = pd.Series([1., np.nan, 2.])
s[s == s]

0    1.0
2    2.0
dtype: float64

Just dropna

s = pd.Series([1., np.nan, 2.])
s.dropna()

0    1.0
2    2.0
dtype: float64

edited Dec 27 '16 at 10:11

answered Dec 27 '16 at 09:48

piRSquared

285,575
57
475
624

1

Are `np.isnan` and `pd.isnull` equivalent? – sergzach Dec 27 '16 at 09:52
@sergzach no, but close. I believe `pd.isnull` checks a few more things for null. – piRSquared Dec 27 '16 at 09:53
4

@sergzach Yes for `NaN`, not for `None`. You cannot use `np.isnan` for object arrays. – ayhan Dec 27 '16 at 09:53
1

Your example is very good but may be not to use `s[s == s]` in practice? It seems ambiguous. – sergzach Dec 27 '16 at 10:09

score 1 · Answer 2 · answered Apr 14 '23 at 07:21

NaN is not equal to NaN; in fact, it's not equal to anything. It's implementation looks similar to the following.¹

class MyNaN(float):
    def __eq__(self, other):
        return False
    def __ne__(self, other):
        return True
    def __repr__(self):
        return 'nan'
    
x = MyNaN()
print(x)        # nan
print(x == x)   # False
print(x != x)   # True

So, one possible way to check if there are NaN values is to check if a value doesn't equal itself. For example, if the aim is to find out which rows contain NaN, instead of checking for equality with NaN, check inequality with itself.

An example:

s = pd.Series(['a', np.nan, 'b'])
x = s[s != s]
y = s[s.isnull()]

x.equals(y)       # True

`np.isnan` vs `pd.isnull`

np.isnan only works for numeric values, while pd.isnull works for all kinds of datatypes.

s = pd.Series(['a', np.nan, 'b'])

s.isnull()        # OK
np.isnan(s)       # TypeError

^{1: Its actual CPython implementation is much more involved. This is only for illustration.}

The difference between comparison to np.nan and isnull()

2 Answers2

`np.isnan` vs `pd.isnull`

Linked

The difference between comparison to np.nan and isnull()

2 Answers2

np.isnan vs pd.isnull

Linked

`np.isnan` vs `pd.isnull`