17

I would like to find where None is found in the dataframe.

pd.DataFrame([None,np.nan]).isnull()
OUT: 
      0
0  True
1  True

isnull() finds both numpy Nan and None values.

I only want the None values and not numpy Nan. Is there an easier way to do that without looping through the dataframe?

Edit: After reading the comments, I realized that in my dataframe in my work also include strings, so the None were not coerced to numpy Nan. So the answer given by Pisdom works.

ConanG
  • 639
  • 1
  • 6
  • 18
  • Afaik, this is not possible in Pandas: Pandas treats `None`s as missing data, and makes them (equivalent to) NaNs. Even by "looping through the dataframe", you won't be able to distinguish `None` from `NaN`. See the [documentation on missing data](https://pandas.pydata.org/pandas-docs/stable/missing_data.html). –  Jul 24 '17 at 01:09
  • 1
    You may want to rethink your problem, and see if you *really* need to distinguish between NaN and `None`. Alternatively, you could introduce a (boolean) column indicating whether a value is `None` or a floating point value (including `NaN`); aka as a mask. –  Jul 24 '17 at 01:17

3 Answers3

18

If you want to get True/False for each line, you can use the following code. Here is an example as a result for the following DataFrame:

df = pd.DataFrame([[None, 3], ["", np.nan]])

df
#      0      1
#0  None    3.0
#1          NaN

How to check None

Available: .isnull()

>>> df[0].isnull()
0     True
1    False
Name: 0, dtype: bool

Available: .apply == or is None

>>> df[0].apply(lambda x: x == None)
0     True
1    False
Name: 0, dtype: bool

>>> df[0].apply(lambda x: x is None)
0     True
1    False
Name: 0, dtype: bool

Available: .values == None

>>> df[0].values == None
array([ True, False])

Unavailable: is or ==

>>> df[0] is None
False

>>> df[0] == None
0    False
1    False
Name: 0, dtype: bool

Unavailable: .values is None

>>> df[0].values is None
False

How to check np.nan

Available: .isnull()

>>> df[1].isnull()
0    False
1     True
Name: 1, dtype: bool

Available: np.isnan

>>> np.isnan(df[1])
0    False
1     True
Name: 1, dtype: bool

>>> np.isnan(df[1].values)
array([False,  True])

>>> df[1].apply(lambda x: np.isnan(x))
0    False
1     True
Name: 1, dtype: bool

Unavailable: is or == np.nan

>>> df[1] is np.nan
False

>>> df[1] == np.nan
0    False
1    False
Name: 1, dtype: bool

>>> df[1].values is np.nan
False

>>> df[1].values == np.nan
array([False, False])

>>> df[1].apply(lambda x: x is np.nan)
0    False
1    False
Name: 1, dtype: bool

>>> df[1].apply(lambda x: x == np.nan)
0    False
1    False
Name: 1, dtype: bool
Keiku
  • 8,205
  • 4
  • 41
  • 44
9

You could use applymap with a lambda to check if an element is None as follows, (constructed a different example, as in your original one, None is coerced to np.nan because the data type is float, you will need an object type column to hold None as is, or as commented by @Evert, None and NaN are indistinguishable in numeric type columns):

df = pd.DataFrame([[None, 3], ["", np.nan]])

df
#      0      1
#0  None    3.0
#1          NaN

df.applymap(lambda x: x is None)

#       0       1
#0   True   False
#1  False   False
Psidom
  • 209,562
  • 33
  • 339
  • 356
  • 1
    This *only* works because you are using an empty string, thus changing the dtype; change the string to any number (integer or float), and the `None` will change into a `NaN`. –  Jul 24 '17 at 01:15
  • @Evert Yes. I added the note that `None` can only exist in *object* type columns. – Psidom Jul 24 '17 at 01:16
1

Q: How check for None in DataFrame / Series

A: isna works but also catches nan. Two suggestions:

  1. Use x.isna() and replace none with nan
  2. If you really care about None: x.applymap(type) == type(None)

I prefer comparing type since for example nan == nan is false. In my case the Nones appeared unintentionally so x[x.isna()] = nan solved the problem.

Example:

x = pd.DataFrame([12, False, 0, nan, None]).T

x.isna()
      0      1      2     3     4
0  False  False  False  True  True

x.applymap(type) == type(None)
       0      1      2      3     4
0  False  False  False  False  True

x
    0      1  2    3     4
0  12  False  0  NaN  None

x[x.isna()] = nan
    0      1  2    3    4
0  12  False  0  NaN  NaN
Hunaphu
  • 589
  • 10
  • 11