65

Is there a way to efficiently implement a rolling window for 1D arrays in Numpy?

For example, I have this pure Python code snippet to calculate the rolling standard deviations for a 1D list, where observations is the 1D list of values, and n is the window length for the standard deviation:

stdev = []
for i, data in enumerate(observations[n-1:]):
    strip = observations[i:i+n]
    mean = sum(strip) / n
    stdev.append(sqrt(250*sum([(s-mean)**2 for s in strip])/(n-1)))

Is there a way to do this completely within Numpy, i.e., without any Python loops? The standard deviation is trivial with numpy.std, but the rolling window part completely stumps me.

I found this blog post regarding a rolling window in Numpy, but it doesn't seem to be for 1D arrays.

Marco Cerliani
  • 21,233
  • 3
  • 49
  • 54
c00kiemonster
  • 22,241
  • 34
  • 95
  • 133
  • 1
    You can also take a look at the [bottleneck project](https://github.com/kwgoodman/bottleneck), it has built in moving average, std, etc. – derchambers May 30 '17 at 00:28

7 Answers7

83

Just use the blog code, but apply your function to the result.

i.e.

numpy.std(rolling_window(observations, n), 1)

where you have (from the blog):

def rolling_window(a, window):
    shape = a.shape[:-1] + (a.shape[-1] - window + 1, window)
    strides = a.strides + (a.strides[-1],)
    return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)
Hooked
  • 84,485
  • 43
  • 192
  • 261
so12311
  • 4,179
  • 1
  • 29
  • 37
  • 1
    Can this be extended to an axis of a n-d array? – Gulzar Jun 27 '21 at 16:23
  • 1
    Also how to use this to create overlap in strides? – Gulzar Jun 27 '21 at 16:37
  • @Gulzar, if you haven't already, check out my answer below. so12311's function actually works on an n-d array with the moving window along the last axis. My function works on an n-d array with the moving window along the first axis. If you need to roll along an intermediary axis, you would need to split the strides and shape and add the extra element at the appropriate location. I don't know what you mean be overlap, but you can probably achieve it with a different stride and shape. Consider asking a new question with the details of what you specifically need. – Leland Hepworth Jul 06 '21 at 14:12
  • 1
    @LelandHepworth Thanks. By overlap I mean for exaple for the array [1, 2, 3, 4, 5] I want some way to get `[1, 2, 3]`, `[2, 3, 4]`, `[3, 4, 5]` for example. – Gulzar Jul 06 '21 at 14:37
  • 1
    @Gulzar, using either version of the function, the following should get you what you want: `rolling_window(np.array([1, 2, 3, 4, 5]), 3)` – Leland Hepworth Jul 06 '21 at 14:45
44

Starting in Numpy 1.20, you can directly get a rolling window with sliding_window_view:

from numpy.lib.stride_tricks import sliding_window_view

sliding_window_view(np.array([1, 2, 3, 4, 5, 6]), window_shape = 3)
# array([[1, 2, 3],
#        [2, 3, 4],
#        [3, 4, 5],
#        [4, 5, 6]])
Xavier Guihot
  • 54,987
  • 21
  • 291
  • 190
  • And to add striding, one can use slicing: `sliding_window_view(np.array([1, 2, 3, 4, 5, 6]), window_shape = 3)[::stride]` I know that `stride_tricks` implements this but the warning messages make me hesitant to use it and this trick handles it well. – Hephaestus Jul 13 '22 at 16:06
14

I tried using so12311's answer listed above on a 2D array with shape [samples, features] in order to get an output array with shape [samples, timesteps, features] for use with a convolution or lstm neural network, but it wasn't working quite right. After digging into how the strides were working, I realized that it was moving the window along the last axis, so I made some adjustments so that the window is moved along the first axis instead:

def rolling_window(a, window_size):
    shape = (a.shape[0] - window_size + 1, window_size) + a.shape[1:]
    strides = (a.strides[0],) + a.strides
    return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)

NOTE: there is no difference in the output if you are only using a 1D input array. In my search this was the first result to get close to what I wanted to do, so I am adding this to help any others searching for a similar answer.

Leland Hepworth
  • 876
  • 9
  • 16
12

With only one line of code...

import pandas as pd

pd.Series(observations).rolling(n).std()
Marco Cerliani
  • 21,233
  • 3
  • 49
  • 54
7

Based on latter answers, here I add code for rolling 1-D numpy arrays choosing window size and window steps frequency.

a = np.arange(50)

def rolling_window(array, window_size,freq):
    shape = (array.shape[0] - window_size + 1, window_size)
    strides = (array.strides[0],) + array.strides
    rolled = np.lib.stride_tricks.as_strided(array, shape=shape, strides=strides)
    return rolled[np.arange(0,shape[0],freq)]

rolling_window(a,10,5)

Output:

array([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9],
       [ 5,  6,  7,  8,  9, 10, 11, 12, 13, 14],
       [10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
       [15, 16, 17, 18, 19, 20, 21, 22, 23, 24],
       [20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
       [25, 26, 27, 28, 29, 30, 31, 32, 33, 34],
       [30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
       [35, 36, 37, 38, 39, 40, 41, 42, 43, 44],
       [40, 41, 42, 43, 44, 45, 46, 47, 48, 49]])
Miguel Gonzalez
  • 706
  • 7
  • 20
3
def moving_avg(x,n):
    mv =  np.convolve(x,np.ones(n)/n,mode='valid')
    return np.concatenate(([np.NaN for k in range(n-1)],mv))
Boris Wang
  • 31
  • 2
0

I needed a rolling window to apply to any intermediate axis of an n-dimensional array, so I extended the code from the already accepted answer and @Miguel Gonzalez. The corresponding code to apply a rolling window to an n-d array along any axis:

def rolling_window(array, window, freq, axis=0):
    shape = array.shape[:axis] + (array.shape[axis] - window_size + 1, window_size) + array.shape[axis+1:]
    strides = array.strides[:axis] + (array.strides[axis],) + array.strides[axis:]
    rolled = np.lib.stride_tricks.as_strided(array, shape=shape, strides=strides)
    return np.take(rolled, np.arange(0,shape[axis],freq), axis=axis)

An example to create a test to assert validity of the function:

    arr = np.random.randint(1, 1000, size=(2,108,21,5))
    arr_windowed = rolling_window_ndimensional(arr, 12, 12, axis=1)

    print(arr.shape)
    print(arr_windowed.shape)
    np.allclose(arr, arr_windowed.reshape(2,-1, 21,5))
Novin Shahroudi
  • 620
  • 8
  • 18