Sum of true values over past n dates in pandas

Question

I have a dataframe of several thousand rows with columns of geography, response_dates and True/False for in_compliance.

df = pd.DataFrame( { 
"geography" : ["Baltimore", "Frederick", "Annapolis", "Hagerstown", "Rockville" , "Salisbury","Towson","Bowie"] , 
"response_date" : ["2018-03-31", "2018-03-30", "2018-03-28", "2018-03-28", "2018-04-02", "2018-03-30","2018-04-07","2018-04-02"],
"in_compliance" : [True, True, False, True, False, True, False, True]})

I want to add a column that represents the number of True values for the most recent four dates in the response_date column, including the response_date for that row. An example of the desired output:

 geography  response_date   in_compliance   Past_4_dates_sum_of_true
Baltimore   2018-03-24  True    1
Baltimore   2018-03-25  False   1
Baltimore   2018-03-26  False   1
Baltimore   2018-03-27  False   1
Baltimore   2018-03-30  False   0
Baltimore   2018-03-31  True    1
Baltimore   2018-04-01  True    2
Baltimore   2018-04-02  True    3
Baltimore   2018-04-03  False   3
Baltimore   2018-04-06  True    3
Baltimore   2018-04-07  True    3
Baltimore   2018-04-08  False   2

I've tried different approaches to groupby and rolling. But I get results that are not what I expect and need.

df.groupby('city').resample('d').sum().fillna(0).groupby('city').rolling(4,min_periods=1).sum()

This was another approach I took:

    df1 = df.groupby(['city']).apply(lambda x: x.set_index('response_date').resample('1D').first())
    df2 = df1.groupby(level=0)['in_compliance']\
         .apply(lambda x: x.shift().rolling(min_periods=1,window=4).count())\
         .reset_index(name='Past_4_dates_sum_of_true')

score 0 · Answer 1 · answered Dec 01 '19 at 08:49

It's much simplier:

df['Past_4_dates_sum_of_true'] = df.rolling(4, min_periods=1)['in_compliance'].sum().astype(int)

Output:

       geography response_date  in_compliance  Past_4_dates_sum_of_true
0   Baltimore    2018-03-24           True                         1
1   Baltimore    2018-03-25          False                         1
2   Baltimore    2018-03-26          False                         1
3   Baltimore    2018-03-27          False                         1
4   Baltimore    2018-03-30          False                         0
5   Baltimore    2018-03-31           True                         1
6   Baltimore    2018-04-01           True                         2
7   Baltimore    2018-04-02           True                         3
8   Baltimore    2018-04-03          False                         3
9   Baltimore    2018-04-06           True                         3
10  Baltimore    2018-04-07           True                         3
11  Baltimore    2018-04-08          False                         2

score 0 · Accepted Answer · answered Dec 01 '19 at 09:39

0

I think you can use rolling with 4days with 4d:

df = df.sort_values(['city','response_date'])
df = df.set_index('response_date')

df['new'] = (df.groupby('city')['in_compliance']
               .rolling('4d',min_periods=1)
               .sum()
               .astype(int)
               .reset_index(level=0, drop=True))
df = df.reset_index()
print (df)
   response_date       city  in_compliance  Past_4_dates_sum_of_true  new
0     2018-03-24  Baltimore           True                         1    1
1     2018-03-25  Baltimore          False                         1    1
2     2018-03-26  Baltimore          False                         1    1
3     2018-03-27  Baltimore          False                         1    1
4     2018-03-30  Baltimore          False                         0    0
5     2018-03-31  Baltimore           True                         1    1
6     2018-04-01  Baltimore           True                         2    2
7     2018-04-02  Baltimore           True                         3    3
8     2018-04-03  Baltimore          False                         3    3
9     2018-04-06  Baltimore           True                         3    1 <-difference because 2018-04-05 missing
10    2018-04-07  Baltimore           True                         3    2
11    2018-04-08  Baltimore          False                         2    2

answered Dec 01 '19 at 09:39

jezrael

822,522
95
1,334
1,252

Thank you -- this works well for sequences of response dates in which there are no response_date gaps. Thank you for noting the difference caused by a gap. Is there a way to get rolling 4d to count back four _dates_ in the df, as opposed to four _days_? – JamesMiller Dec 02 '19 at 03:26
@JamesMiller - do you think like your last paragraph? I find only simplify your solution `df1 = df.groupby(['city']).apply(lambda x: x.set_index('response_date').resample('1D').first())` to `df1 = df.groupby(['city']).resample('1D').first()` (before is created DatetimeIndex) – jezrael Dec 02 '19 at 06:30
What is the best approach to creating a DatetimeIndex? I've tried it a couple ways and get error that response_date is an instance of 'Int64Index'. – JamesMiller Dec 02 '19 at 17:56
@JamesMiller use `df = df.set_index('response_date') ` – jezrael Dec 02 '19 at 19:14
Thanks @jezrael. I ended up adding a column of the number of business days between the response date and when the review started. Then I modified your code slightly to get the in_compliance sum for the previous four rows, irrespective of gaps in dates. This [post](https://stackoverflow.com/questions/43787059/how-to-compute-cumulative-sum-of-previous-n-rows-in-pandas) assisted. – JamesMiller Dec 03 '19 at 00:34

Sum of true values over past n dates in pandas

2 Answers2