rolling function in python ignoring nans

Question

https://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.rolling_quantile.html

I cant not see how to best ignore NaNs in the rolling percentile function. Would anyone know?

seriestest = pd.Series([1, 5, 7, 2, 4, 6, 9, 3, 8, 10])

and insert nans

seriestest2 = pd.Series([1, 5, np.NaN, 2, 4, np.nan, 9, 3, 8, 10])

Now, on the first series, I get expected output, using:

seriestest.rolling(window = 3).quantile(.5)

But, I wish to do the same and ignore NaNs on the test2 series.

seriestest2.rolling(window = 3).quantile(.5)

Gives:

0    NaN
1    NaN
2    NaN
3    NaN
4    NaN
5    NaN
6    NaN
7    NaN
8    8.0
9    8.0
dtype: float64

But I think it gives something like this if we can parse a skipna=True, which doesn't work for me:

0    NaN
1    NaN
2    5.0
3    2.0
4    4.0
5    4.0
6    4.0
7    3.0
8    8.0
9    8.0
dtype: float64

What functionality do you expect when there's a `NaN` in the window? — Prune, Sep 10 '18 at 17:58
@Prune I wish to ignore the NaN and pretend it isn't there, i.e. work off all the integers in that window period — Junaid Mohammad, Sep 10 '18 at 18:15
"Pretend it isn't there" comes in various forms. For instance, when you get to the first full window `(1, 5, NaN)`, do you want the visible two values only, or do you want to infill with the next available value of 2? — Prune, Sep 10 '18 at 18:26

busybear · Accepted Answer · 2018-09-10T18:48:05.200

6

The issue is that having nan values will give you less than the required number of elements (3) in your rolling window. You can define the minimum number of valid observations with rolling to be less by setting the min_periods parameter.

seriestest2.rolling(window=3, min_periods=1).quantile(.5)

Alternatively, if you simply want to replace nan values, with say 0, you can use fillna:

seriestest2.fillna(value=0).rolling(window=3).quantile(.5)

edited Sep 10 '18 at 18:48

answered Sep 10 '18 at 18:14

busybear

10,194
1
25
42

This doesn't match the values the OP expects, though, so the OP may have something else in mind. – DSM Sep 10 '18 at 18:24
Unfortunately, I'm unclear on how those numbers were obtained. This seems to fit the description of what was expected however. – busybear Sep 10 '18 at 18:47
Thanks @busybear. Ill give it a go.. I think you have solved what I was aiming to describe – Junaid Mohammad Sep 10 '18 at 19:28
Also filling with 0 (or any other constant value) may modify the result in the general case (as opposed to ignoring nan values in the window when calculating the statistic) – Yuri Feldman Oct 19 '22 at 14:53

rolling function in python ignoring nans

1 Answers1

Linked