Rolling Gradient for Pandas Dataframe column

Question

How can I create a column in a pandas dataframe with is the gradient of another column?

I want the gradient to be run over a rolling window, so only 4 data points are assessed at one time.

I am assuming it is something like:

df['Gradient'] = np.gradient(df['Yvalues'].rolling(center=False,window=4))

However this gives error:

    raise ValueError('Length of values does not match length of ' 'index')
ValueError: Length of values does not match length of index

Any ideas?

Thank you!!

Please provide sample data or statistics about your data set, i.e. number of data points etc. — Bryce Ramgovind, Jan 04 '18 at 07:04
not many data points. eg a column with 100 150 209 319 400 then an adjacent column would have a rolling window of the gradient — cwse, Jan 04 '18 at 07:08

score 0 · Answer 1 · edited Apr 16 '22 at 13:58

0

From the given information, it can be seen that you haven't provided an aggregation function to your rolling window.

df['Gradient'] = np.gradient(
    df['Yvalues']
    .rolling(center=False, window=4)
    .mean()
)

or

df['Gradient'] = np.gradient(
    df['Yvalues']
    .rolling(center=False, window=4)
    .sum()
)

You can read more about rolling functions at this website.

edited Apr 16 '22 at 13:58

Jaroslav Bezděk

6,967
6
29
46

answered Jan 04 '18 at 07:20

Bryce Ramgovind

3,127
10
41
72

Hi @Bryce Ramgovind. I dont want an aggregate function... the gradient should be the function. I am after a rolling window of 4 rows at a time. – cwse Jan 04 '18 at 08:52

score 0 · Answer 2 · edited Apr 16 '22 at 13:39

I think I found the solution. Though it's probably not the most efficient..

class lines(object):
    def __init__(self):
        pass

    def date_index_to_integer_axis(self, dateindex):
        d = [d.date() for d in dateindex]
        days = [(d[x] - d[x-1]).days for x in range(0,len(d))]
        axis = np.cumsum(days)
        axis = [x - days[0] for x in axis]
        return axis

    def roll(self, Xvalues, Yvalues, w):  # Rollings Generator Function # https://stackoverflow.com/questions/231767/what-does-the-yield-keyword-do-in-python
        for i in range(len(Xvalues) + 1 - w):
            yield Xvalues[i:i + w], Yvalues[i:i + w]

    def gradient(self,Xvalues,Yvalues):
        #Uses least squares method.
        #Returns the gradient of two array vectors (https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.linalg.lstsq.html)
        A = np.vstack([Xvalues, np.ones(len(Xvalues))]).T
        m, c = np.linalg.lstsq(A, Yvalues)[0]
        return m,c

    def gradient_column(self, data, window):
        """ Takes in a single COLUMN EXTRACT from a DATAFRAME (with  associated "DATE" index) """
        vars = variables()

        #get "X" values
        Xvalues = self.date_index_to_integer_axis(data.index)
        Xvalues = np.asarray(Xvalues,dtype=np.float)
        #get "Y" values
        Yvalues = np.asarray([val for val in data],dtype=np.float)
        Yvalues = np.asarray(Yvalues,dtype=np.float)

        #calculate rolling window "Gradient" ("m" in Y = mx + c)
        Gradient_Col = [self.gradient(sx,sy)[0] for sx,sy in self.roll(Xvalues,Yvalues, int(window))]
        Gradient_Col = np.asarray(Gradient_Col,dtype=np.float)

        nan_array = np.empty([int(window)-1])
        nan_array[:] = np.nan
        #fill blanks at the start of the "Gradient_Col" so it is the same length as the original Dataframe (its shorter due to WINDOW)
        Gradient_Col = np.insert(Gradient_Col, 0, nan_array)

        return Gradient_Col


df['Gradient'] = lines.gradient_column(df['Operating Revenue'],window=4)

Rolling Gradient for Pandas Dataframe column

2 Answers2