pandas: group by ffill does not apply fill in correct order

Question

i am facing issue with group by ffill. It does not seem to apply forward fill in correct order Here is my starting data

   group       date  stage_2
0      A 2014-01-01      NaN
1      A 2014-01-03      NaN
2      A 2014-01-04      NaN
3      A 2014-01-05      1.0
4      B 2014-01-02      NaN
5      B 2014-01-06      NaN
6      B 2014-01-10      NaN
7      C 2014-01-03      1.0
8      C 2014-01-05      3.0
9      C 2014-01-08      NaN
10     C 2014-01-09      NaN
11     C 2014-01-10      NaN
12     C 2014-01-11      NaN
13     D 2014-01-01      NaN
14     D 2014-01-03      NaN
15     D 2014-01-04      NaN
16     E 2014-01-04      1.0
17     E 2014-01-06      3.0
18     E 2014-01-07      4.0
19     E 2014-01-08      NaN
20     E 2014-01-09      NaN
21     E 2014-01-10      NaN
22     F 2014-01-08      NaN

After applying the ffill method this is what i get

df['stage_2'] = df.groupby('group')['stage_2'].ffill()

I am expecting a different value at index 9 through 12 and 21

   group       date  stage_2
0      A 2014-01-01      NaN
1      A 2014-01-03      NaN
2      A 2014-01-04      NaN
3      A 2014-01-05      1.0
4      B 2014-01-02      NaN
5      B 2014-01-06      NaN
6      B 2014-01-10      NaN
7      C 2014-01-03      1.0
8      C 2014-01-05      3.0
9      C 2014-01-08      1.0
10     C 2014-01-09      NaN
11     C 2014-01-10      NaN
12     C 2014-01-11      NaN
13     D 2014-01-01      NaN
14     D 2014-01-03      NaN
15     D 2014-01-04      NaN
16     E 2014-01-04      1.0
17     E 2014-01-06      3.0
18     E 2014-01-07      4.0
19     E 2014-01-08      4.0
20     E 2014-01-09      4.0
21     E 2014-01-10      NaN
22     F 2014-01-08      NaN

I cannot reproduce these results, pandas `0.23.4`, Python 3.6.5 — user3483203, Dec 09 '18 at 00:56

ayorgo · Accepted Answer · 2018-12-09T04:17:10.567

0

The only way I can reproduce this is by putting in non-ASCII characters e.g. Cyrillic С and Е into the group column at indexes 9-12 and 21 respectively.

EDIT

OK, most likely you're using pandas v0.23.0 which had a bug (fixed in future versions, at least in v0.23.4) that makes .ffill() give the exact output you posted. So please upgrade your pandas.

edited Dec 09 '18 at 04:17

answered Dec 09 '18 at 00:23

ayorgo

2,803
2
25
35

thanks for your response. i have put my code snippet at https://github.com/vikashya/work/blob/master/Rolling%20difference-Copy2.ipynb i dont have non ascii characters in my file – Jan.V Dec 09 '18 at 00:48

pandas: group by ffill does not apply fill in correct order

1 Answers1