extracting an expanding window (numpy strided) of a DataFrame

Question

(related to this answer)

Given a df, I was expecting to get the results of df.expanding() and to perform some multivariate operations on this (operations involving several columns of df simultaneously over an expanding window of rows) using .apply(). It turns out this is not possible.

So, much as in the answer linked above, I need to use numpy.as_strides of df. Except, in contrast to the question linked above, using strides to get an expanding views of my df, instead of rolling ones (an expanding window has a fixed left side and the right side moves to the right incrementally).

consider this df:

import numpy
import pandas


df = pandas.DataFrame(numpy.random.normal(0, 1, [100, 2]), columns=['size_A', 'size_B']).cumsum(axis=0)

consider this code to extract a rolling window of W rows of that df (this comes from the answer above):

def get_sliding_window(df, W):
    a = df.values                 
    s0,s1 = a.strides
    m,n = a.shape
    return numpy.lib.stride_tricks\
               .as_strided(a,shape=(m-W+1,W,n),strides=(s0,s0,s1))

roll_window = get_sliding_window(df, W = 3)
roll_window[2] 

Now I want to modify get_sliding_window to get it to return an expanding window of a df (instead of a rolling one):

def get_expanding_window(df):
    a = df.values                 
    s0,s1 = a.strides
    m,n = a.shape
    out = numpy.lib.stride_tricks\
               .as_strided(a, shape=(m,m,n),strides=(s0,s0,s1))
    return out

expg_window = get_expanding_window(df)
expg_window[2]

But I'm not using the arguments of as_strided correctly: I can't seem to be getting the right matrices --that would be something like:

[df.iloc[0:1].values ,df.iloc[0:2].values, df.iloc[0:3].values,...]  

Edit:

In a comment @ThomasKühn proposes to use list comprehension. This would solve the problem, but too slowly. What is the cost?

One a vector valued function, we can compare the cost of list comprehension with .expand(). It ain't small:

numpy.random.seed(123)
df = pandas.DataFrame((numpy.random.normal(0, 1, 10000)), columns=['Value'])
%timeit method_1 = numpy.array([df.Value.iloc[range(j + 1)].sum() for j in range(df.shape[0])])

gives:

6.37 s ± 219 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

comparing to .expanding():

%timeit method_2 = df.expanding(0).apply(lambda x: x.sum())

which gives:

35.5 ms ± 356 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Finally, there are more details as to the problem I am trying to solve in the comments to this question.

Answer 1

I wrote a few functions that are all supposed to do the same thing, but need different amounts of time to complete the task:

import timeit
import numba as nb

x = np.random.normal(0,1,(10000,2))
def f1():
    res = [np.sum(x[:i,0] > x[i,1]) for i in range(x.shape[0])]
    return res

def f2():
    buf = np.empty(x.shape[0])
    res = np.empty(x.shape[0])
    for i in range(x.shape[0]):
        buf[:i] = x[:i,0] > x[i,1]
        res[i] = np.sum(buf[:i])
    return res

def f3():
    res = np.empty(x.shape[0])
    for i in range(x.shape[0]):
        res[i] = np.sum(x[:i,0] > x[i,1])
    return res


@nb.jit(nopython=True)
def f2_nb():
    buf = np.empty(x.shape[0])
    res = np.empty(x.shape[0])
    for i in range(x.shape[0]):
        buf[:i] = x[:i,0] > x[i,1]
        res[i] = np.sum(buf[:i])
    return res

@nb.jit(nopython=True)
def f3_nb():
    res = np.empty(x.shape[0])
    for i in range(x.shape[0]):
        res[i] = np.sum(x[:i,0] > x[i,1])
    return res

##checking that all functions give the same result:
print('checking correctness')
print(np.all(f1()==f2()))
print(np.all(f1()==f3()))
print(np.all(f1()==f2_nb()))
print(np.all(f1()==f3_nb()))

print('+'*50)
print('performance tests')
print('f1()')        
print(min(timeit.Timer(
    'f1()',
    setup = 'from __main__ import f1,x',
).repeat(7,10)))

print('-'*50)
print('f2()')
print(min(timeit.Timer(
    'f2()',
    setup = 'from __main__ import f2,x',
).repeat(7,10)))

print('-'*50)
print('f3()')
print(min(timeit.Timer(
    'f3()',
    setup = 'from __main__ import f3,x',
).repeat(7,10)))

print('-'*50)
print('f2_nb()')
print(min(timeit.Timer(
    'f2_nb()',
    setup = 'from __main__ import f2_nb,x',
).repeat(7,10)))

print('-'*50)
print('f3_nb()')
print(min(timeit.Timer(
    'f3_nb()',
    setup = 'from __main__ import f3_nb,x',
).repeat(7,10)))

As you can see, the differences are not big, but there are some differences in performance. The last two functions are just 'duplicates' of earlier ones, but using numba optimisation. The results of the speed tests are

checking correctness
True
True
True
True
++++++++++++++++++++++++++++++++++++++++++++++++++
performance tests
f1()
2.02294262702344
--------------------------------------------------
f2()
3.0964318679762073
--------------------------------------------------
f3()
1.9573561699944548
--------------------------------------------------
f2_nb()
1.3796060049789958
--------------------------------------------------
f3_nb()
0.48667875200044364

As you can see, the differences are not terribly big, but between the slowest and the fastest function, the speedup is anyway about 6 times. Hope this helps.