Is it possible to use panda's expanding function to calculate the coefficient of a polynomial regression using several columns of the window object?
I have a data frame which has two columns, a predictor and a response. I want to use pandas' expanding() function to calculate the corresponding coefficients of a second order polynomial regression for each expanding pair of series. For each row I would like to get the updated coefficients from the regression applied to all previous rows.
import pandas as pd
import numpy as np
def func1(df):
# some processing
return np.polyfit(df['Input'], df['Response'], 2)
def func2(x, y):
# some processing
return np.polyfit(x, y, 2)
np.random.seed(0)
df = pd.DataFrame(np.random.rand(10, 2).round(2),
columns=['Input', 'Response'])
df[['Coef1', 'Coef2', 'Coef3']] = df.expanding(min_periods=3).apply(func)
I'd like the output to look like this:
>>> df
Input Response Coef1 Coef2 Coef3
0 0.63 0.23 NaN NaN NaN
1 0.45 0.11 NaN NaN NaN
2 0.17 0.71 NaN NaN NaN
3 0.17 0.32 0.19 0.54 0.50
4 0.65 0.99 0.48 0.23 0.60
5 0.21 0.54 0.71 0.89 0.97
6 0.63 0.73 0.22 0.05 0.80
7 0.54 0.23 0.87 0.01 0.25
8 0.33 0.06 0.18 0.96 0.03
9 0.18 0.72 0.13 0.38 0.13
My different trials has led to two types of error. If I use the function that uses the dataframe as a parameter such as in df[['Coef1', 'Coef2', 'Coef3']] = df.expanding(min_periods=3).apply(func1)), I get KeyError: 'Input'.
If I use the second function where I extract the parameters before df['Coef1', 'Coef2', 'Coef3'] = df.expanding(min_periods=3).apply(lambda x: func2(x['Input'], x['Output'])), I get
DataError: No numeric types to aggregate
However, If I try for instance df.expanding().cov(pairwise=True) it shows that calculation can be performed on the different columns of the object returned by expanding.
There's a similar question here: Apply expanding function on dataframe. However, the solution consisting in calling expanding() in the function does not seem to apply in this case.
I would appreciate any pointers or suggestion.
I found a package that does that with numpy so it inspired me to do it manually:
def func_np(df):
length = len(df)
if length == 1:
return [[0], [0], [0]]
coef1, coef2, coef3 = [], [], []
x = df['A'].to_numpy() # This is the predictor column
y = df['B'].to_numpy() # This is the response column
for step in range(1, length + 1):
weights = np.polyfit(x[: step], y[: step], 2) # 2 is the polynomial's order
coef1.append(weights[0])
coef2.append(weights[1])
coef3.append(weights[2])
# Note that coef1, coef2, coef3 correspond to the polynomial terms from highest to lowest
# It is easier to return a data frame, so that we can reassign the result to the initial one
return pd.DataFrame({'Coef1': coef1, 'Coef2': coef2, 'Coef3': coef3})
I wanted to do it with Numba to speed up the execution but it does not recognize the np.polyfit function. Also I have not found a neat way to assign back the results to the initial data frame. That is why I am still interested in seeing a simple and more "pythonic" solution with expanding()
