3

Let's say that I start with this dataframe

d = {'price': [10, 12, 8, 12, 14, 18, 10, 20],
     'volume': [50, 60, 40, 100, 50, 100, 40, 50]}
df = pd.DataFrame(d)
df['a_date'] = pd.date_range('01/01/2018',
                             periods=8,
                             freq='W')

df
    price   volume  a_date
0   10      50      2018-01-07
1   12      60      2018-01-14
2   8       40      2018-01-21
3   12      100     2018-01-28
4   14      50      2018-02-04
5   18      100     2018-02-11
6   10      40      2018-02-18
7   20      50      2018-02-25

Now, I would like to resample/groupby in such a way that the data is aggregated on time intervals of roughly 10 days, but with pre-defined start and end dates, which fall on the 10th, 20th and last day of the month, such as:

2018-01-01 to 2018-01-10
2018-01-11 to 2018-01-20
2018-01-21 to 2018-01-31
2018-02-01 to 2018-02-10
2018-02-11 to 2018-02-20
2018-02-21 to 2018-02-28

and the results would be, in case of summing across the intervals:

             price  volume  
a_date
2018-01-10   10     50      
2018-01-20   12     60      
2018-01-31   20     140     
2018-02-10   14     50      
2018-02-20   28     140     
2018-02-28   20     50      

The closest that I can get to this is doing df.resample('10D', on='a_date').sum() but clearly I need something more customized as interval. I would be happy with just passing an array of intervals, but I don't think that's possible.

I've tried, as an experiment:

td = pd.to_datetime('2018-01-10') - pd.to_datetime('2018-01-01')
df.resample(td, on='a_date').sum()

but the pandas.Timedelta does not keep information on the specific dates.

EDIT:

A different dataframe to test the first day of the month:

d = {'price': np.arange(20)+1,
    'volume': np.arange(20)+5}
df = pd.DataFrame(d)
df['a_date'] = pd.date_range('01/01/2018',
                             periods=20,
                             freq='D')

applying the accepted answer gives (the first day is not taken into account):

      a_date  price  volume
0 2018-01-10     54      90
1 2018-01-20    155     195

compare with (for the first interval 2018-01-01 to 2018-01-10):

df.iloc[:10].sum()

price     55
volume    95
dtype: int64
duff18
  • 672
  • 1
  • 6
  • 19

1 Answers1

1

Try:

from pandas.tseries.offsets import MonthEnd

bins = []
end = df["a_date"].max()
current = df["a_date"].min()
current = pd.Timestamp(year=current.year, month=current.month, day=1)
while True:
    bins.append(current)
    bins.append(current + pd.Timedelta(days=9))
    bins.append(current + pd.Timedelta(days=19))
    bins.append(current + MonthEnd())
    if bins[-1] > end:
        break
    current = bins[-1] + pd.Timedelta(days=1)

x = (df.groupby(pd.cut(df["a_date"], bins)).sum()).reset_index()
x["a_date"] = x["a_date"].cat.categories.right
print(x[~(x.price.eq(0) & x.volume.eq(0))])

Prints:

      a_date  price  volume
0 2018-01-10     10      50
1 2018-01-20     12      60
2 2018-01-31     20     140
4 2018-02-10     14      50
5 2018-02-20     28     140
6 2018-02-28     20      50

EDIT: Adjusted bins:

from pandas.tseries.offsets import MonthEnd

end = df["a_date"].max()
current = df["a_date"].min()
bins = [
    pd.Timestamp(year=current.year, month=current.month, day=1) - MonthEnd(),
]
current = bins[-1]
while True:
    bins.append(bins[-1] + pd.Timedelta(days=10))
    bins.append(bins[-1] + pd.Timedelta(days=10))
    bins.append(current + MonthEnd())
    if bins[-1] > end:
        break
    current = bins[-1]

x = (df.groupby(pd.cut(df["a_date"], bins)).sum()).reset_index()
x["a_date"] = x["a_date"].cat.categories.right
print(x[~(x.price.eq(0) & x.volume.eq(0))])

Prints:

      a_date  price  volume
0 2018-01-10     55      95
1 2018-01-20    155     195
Andrej Kesely
  • 168,389
  • 15
  • 48
  • 91
  • Looks great thanks! I see that the last line removes "empty" rows. Is this something that can be applied also to a generic dataframe with different dates? – duff18 Jul 22 '21 at 16:53
  • @duff18 Yes, of course..it's boolean indexing. It removes all rows were price **and** volume is equal to zero. – Andrej Kesely Jul 22 '21 at 16:56
  • small problem, the first day of each month is not included in the first interval of the month. probably `pandas.IntervalIndex` as bins is the way to go – duff18 Jul 22 '21 at 17:26
  • @duff18 Can you provide small input where the result is wrong + expected output? Probably the construction of bins needs to be adjusted. – Andrej Kesely Jul 22 '21 at 17:28
  • Sure, I've added it in the post – duff18 Jul 22 '21 at 17:37