Interactions between dummies variables in python

Question

I'm trying to understand how can I address columns after using get_dummies. For example, let's say I have three categorical variables. first variable has 2 levels. second variable has 5 levels. third variable has 2 levels.

df=pd.DataFrame({"a":["Yes","Yes","No","No","No","Yes","Yes"], "b":["a","b","c","d","e","a","c"],"c":["1","2","2","1","2","1","1"]})

I created dummies for all three variable in order to use them in sklearn regression in python.

df1 = pd.get_dummies(df,drop_first=True)

Now I want to create two interactions (multiplication): bc , ba

how can I create the multiplication between each dummies variable to another one without using their specific names like that:

df1['a_yes_b'] = df1['a_Yes']*df1['b_b']
df1['a_yes_c'] = df1['a_Yes']*df1['b_c']
df1['a_yes_d'] = df1['a_Yes']*df1['b_d']
df1['a_yes_e'] = df1['a_Yes']*df1['b_e']

df1['c_2_b'] = df1['c_2']*df1['b_b']
df1['c_2_c'] = df1['c_2']*df1['b_c']
df1['c_2_d'] = df1['c_2']*df1['b_d']
df1['c_2_e'] = df1['c_2']*df1['b_e']

Thanks.

jezrael · Accepted Answer · 2017-03-23T07:57:11.270

You can use loops for creating new columns, for filtering column names is possible use filtering by boolean indexing and str.startswith:

a = df1.columns[df1.columns.str.startswith('a')]
b = df1.columns[df1.columns.str.startswith('b')]
c = df1.columns[df1.columns.str.startswith('c')]

for col1 in b:
    for col2 in a:
        df1[col2 + '_' + col1.split('_')[1]] = df1[col1].mul(df1[col2])

for col1 in b:
    for col2 in c:
        df1[col2 + '_' + col1.split('_')[1]] = df1[col1].mul(df1[col2])
print (df1)

   a_Yes  b_b  b_c  b_d  b_e  c_2  a_Yes_b  a_Yes_c  a_Yes_d  a_Yes_e  c_2_b  \
0      1    0    0    0    0    0        0        0        0        0      0   
1      1    1    0    0    0    1        1        0        0        0      1   
2      0    0    1    0    0    1        0        0        0        0      0   
3      0    0    0    1    0    0        0        0        0        0      0   
4      0    0    0    0    1    1        0        0        0        0      0   
5      1    0    0    0    0    0        0        0        0        0      0   
6      1    0    1    0    0    0        0        1        0        0      0   

   c_2_c  c_2_d  c_2_e  
0      0      0      0  
1      0      0      0  
2      1      0      0  
3      0      0      0  
4      0      0      1  
5      0      0      0  
6      0      0      0

But if a and b have only one columns (in sample yes, in real data maybe) use: filter, mul, squeeze and concat:

a = df1.filter(regex='^a')
b = df1.filter(regex='^b')
c = df1.filter(regex='^c')

dfa = b.mul(a.squeeze(), axis=0).rename(columns=lambda x: a.columns[0] + x[1:])
dfc = b.mul(c.squeeze(), axis=0).rename(columns=lambda x: c.columns[0] + x[1:])

df1 = pd.concat([df1, dfa, dfc], axis=1)
print (df1)
   a_Yes  b_b  b_c  b_d  b_e  c_2  a_Yes_b  a_Yes_c  a_Yes_d  a_Yes_e  c_2_b  \
0      1    0    0    0    0    0        0        0        0        0      0   
1      1    1    0    0    0    1        1        0        0        0      1   
2      0    0    1    0    0    1        0        0        0        0      0   
3      0    0    0    1    0    0        0        0        0        0      0   
4      0    0    0    0    1    1        0        0        0        0      0   
5      1    0    0    0    0    0        0        0        0        0      0   
6      1    0    1    0    0    0        0        1        0        0      0   

   c_2_c  c_2_d  c_2_e  
0      0      0      0  
1      0      0      0  
2      1      0      0  
3      0      0      0  
4      0      0      1  
5      0      0      0  
6      0      0      0

score 0 · Answer 2 · edited May 23 '17 at 11:46

0

You can convert dataframe column into numpy array and then multiply it accordingly. here's the link where you can find methos to do that:

Convert Select Columns in Pandas Dataframe to Numpy Array

edited May 23 '17 at 11:46

Community

1
1

answered Mar 23 '17 at 07:49

Tarun

150
3
13

score 0 · Answer 3 · answered Feb 21 '21 at 21:51

This solves your problem:

def get_design_with_pair_interaction(data, group_pair):
    """ Get the design matrix with the pairwise interactions
    
    Parameters
    ----------
    data (pandas.DataFrame):
       Pandas data frame with the two variables to build the design matrix of their two main effects and their interaction
    group_pair (iterator):
       List with the name of the two variables (name of the columns) to build the design matrix of their two main effects and their interaction
    
    Returns
    -------
    x_new (pandas.DataFrame):
       Pandas data frame with the design matrix of their two main effects and their interaction
    
    """
    x = pd.get_dummies(data[group_pair])
    interactions_lst = list(
        itertools.combinations(
            x.columns.tolist(),
            2,
        ),
    ) 
    x_new = x.copy()
    for level_1, level_2 in interactions_lst:
        if level_1.split('_')[0] == level_2.split('_')[0]:
            continue
        x_new = pd.concat(
            [
                x_new,
                x[level_1] * x[level_2]
            ],
            axis=1,
        )
        x_new = x_new.rename(
            columns = {
                0: (level_1 + '_' + level_2)
            }
        )
    return x_new

Interactions between dummies variables in python

3 Answers3