39

What is python's equivalent of R's NA?

To be more specific: R has NaN, NA, NULL, Inf and -Inf. NA is generally used when there is missing data. What is python's equivalent?

How libraries such as numpy and pandas handle missing values?

How does scikit-learn handle missing values?

Is it different for python 2.7 and python 3?

power
  • 1,680
  • 3
  • 18
  • 30

3 Answers3

28

nan in numpy is handled well with many functions: 

>>> import numpy as np
>>> a = [1, np.nan, 2, 3]
>>> np.nanmean(a)
2.0
>>> np.nansum(a)
6.0
>>> np.isnan(a)
array([False,  True, False, False], dtype=bool)
N1B4
  • 3,377
  • 1
  • 21
  • 24
13

Scikit-learn doesn't handle missing values currently. For most machine learning algorithms, it is unclear how to handle missing values, and so we rely on the user of handling them prior to giving them to the algorithm. Numpy doesn't have a "missing" value. Pandas uses NaN, but inside numeric algorithms that might lead to confusion. It is possible to use masked arrays, but we don't do that in scikit-learn (yet).

Andreas Mueller
  • 27,470
  • 8
  • 62
  • 74
8

for pandas take a look at this.

http://pandas.pydata.org/pandas-docs/dev/missing_data.html

pandas uses NaN. You can test for null values using isnull() or not null(), drop them from a data frame using dropna() etc. The equivalent for datetime objects is NaT

JAB
  • 12,401
  • 6
  • 45
  • 50
  • 3
    It might be worth noting that integer pandas `Series` (or column) must have values. There is no way to represent a missing value in an integer series; the usual alternative being to upconvert to a floating point type that has `NaN`. – Paul Feb 22 '15 at 04:03