apply a recursive function to multiple pandas dataframe column using groupby

Question

I want to implement exponential filter.

column "value" is unfiltered value
column "alfa" is current weight of the filter
column "value_filtered" is output, i.e. filtered value.

I need to use it in combination with pandas groupby. pseudo code:

value_filtered[0] = value[0];
for i>0: value_filtered[i] = (1-alfa[i])* value_filtered[i-1] + alfa[i] *value[i]

Example is below:

data_out = [['group1', 0, 1,1],['group1', 0.5, 2, 1.5],['group1', 0.6, 3, 2.4],  ['group2', 0, 10, 10],['group2', 0.5, 20, 15],['group2', 0.6, 30, 24], ]

df_out = pd.DataFrame(data_out, columns=['group', 'alfa', 'value', 'value_filtered'])

df_out

I looked to the post below, but there the filter weight is static, while in my case it is defined in column "alfa" and does not work with groupby. apply a recursive function to a pandas dataframe column

Ynjxsjmh · Answer 1 · 2023-03-27T11:54:19.750

You can define a customize function to do your recursive logic in groupby.apply

def filter_value(g):
    values_filtered = [g['value'].iloc[0]] * len(g)
    alfa = g['alfa'].tolist()
    value = g['value'].tolist()

    for i in range(1, len(g)):
        values_filtered[i] = ((1-alfa[i]) * values_filtered[i-1]
                              + alfa[i] * value[i])

    return g.assign(out=values_filtered)

out = df.groupby('group', group_keys=False).apply(filter_value)

print(out)

    group  alfa  value  value_filtered   out
0  group1   0.0      1             1.0   1.0
1  group1   0.5      2             1.5   1.5
2  group1   0.6      3             2.4   2.4
3  group2   0.0     10            10.0  10.0
4  group2   0.5     20            15.0  15.0
5  group2   0.6     30            24.0  24.0

score 0 · Answer 2 · answered Mar 27 '23 at 11:45

You can use custom generator for the task:

def my_func(col_alfa, col_value):
    prev = None
    for a, v in zip(col_alfa, col_value):
        if prev is None:
            prev = v
        else:
            prev = (1 - a) * prev + a * v
        yield prev


df["value_filtered_2"] = (
    df.groupby("group")
    .apply(lambda x: pd.Series(my_func(x["alfa"], x["value"]), index=x.index))
    .droplevel(0)
)
print(df)

Prints:

    group  alfa  value  value_filtered  value_filtered_2
0  group1   0.0      1             1.0               1.0
1  group1   0.5      2             1.5               1.5
2  group1   0.6      3             2.4               2.4
3  group2   0.0     10            10.0              10.0
4  group2   0.5     20            15.0              15.0
5  group2   0.6     30            24.0              24.0

it works for me. it fails though on a dataframe that contains only one group. " ValueError: Cannot remove 1 levels from an index with 1 levels: at least one level must be left." But I can make a workaround to come around this limitation. — alex, Mar 27 '23 at 13:41

apply a recursive function to multiple pandas dataframe column using groupby

2 Answers2