Removing inf/nan values from Pandas

Question

I'm aware that there are several posts about this, but none of the solutions seem to work and I can't figure out what I'm doing wrong.

My dataframe has data with inf values.

print [x for x in train_x['meh'] if not np.isfinite(x)]

returns

[inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf]

I tried

train_x.replace([numpy.inf, -numpy.inf], numpy.nan)

But this does not convert the inf to nan.

Next I tried

train_x[np.isinf(train_x)] = np.NaN

... which does convert the inf to nan but I'm not able to drop the nan rows even with train_x.dropna()

Essentially, I need to drop the inf and nan rows.

You have to specify `inplace=True` in both cases (`replace` and `dropna`). — IanS, Oct 09 '17 at 15:03
or assign it back `train_x=train_x.replace([numpy.inf, -numpy.inf], numpy.nan)` — BENY, Oct 09 '17 at 15:12

score 0 · Answer 1 · answered Oct 09 '17 at 15:16

0

Seems like you can just use boolean indexing

train_x[np.isfinite(train_x) & train_x.notnull()]

You actually don't even need the train_x.notnull().

answered Oct 09 '17 at 15:16

Ted Petrou

59,042
19
131
136

score 0 · Answer 2 · answered Oct 09 '17 at 16:30

I've created a DataFrame using

df = pd.DataFrame({'val1': [1.1, 3.3, 2.2, np.inf], 'val2': [2.2, 6.6, np.inf, 4.4]})

which results in

       val1      val2
0  1.100000  2.200000
1  3.300000  6.600000
2  2.200000       inf
3       inf  4.400000

If you have inf in more than one column you can change it to NaN by using

df.applymap(lambda x: np.nan if np.isinf(x) else x)

And the result is

   val1  val2
0   1.1   2.2
1   3.3   6.6
2   2.2   NaN
3   NaN   4.4

If you have this issue on only one column (in this example, on column val1) you can do

df.val1.apply(lambda x: np.nan if np.isinf(x) else x)

Removing inf/nan values from Pandas

2 Answers2