Join merge multiple dataframes

Question

My data is like this:

Name    test1   test2    test3     Count
Emp1    X,Y      A       a1,a2      1
Emp2    X       A,B,C    a3         2
Emp3    Z        C       a4,a5,a6   3

To split test1 and test2 cells with multiple values to individual rows and merged them together.

    df2 =  df.test1.str.split(',').apply(pd.Series)
    df2.index =  df.set_index(['Name', 'Count']).index
    df2=df2.stack().reset_index(['Name', 'Count'])
    df3 = df.test2.str.split(',').apply(pd.Series)
    df3.index = df.set_index(['Name', 'Count']).index
    df3=df3.stack().reset_index(['Name', 'Count'])

    df2.merge(df3,on=['Name', 'Count'],how='outer')

The out of code is :

Out[132]: 
   Name  Count 0_x 0_y
0  Emp1      1   X   A
1  Emp1      1   Y   A
2  Emp2      2   X   A
3  Emp2      2   X   B
4  Emp2      2   X   C
5  Emp3      3   Z   C

Code to split Test3 with multiple values to individual rows

    df4.index = df.set_index(['Name', 'Count']).index
    df4=df4.stack().reset_index(['Name', 'Count'])

Can anyone help me, how to multi-join Test3 with test2 and test1 Like i merged Test1 and Test in above code?

score 2 · Answer 1 · answered Sep 22 '18 at 02:47

2

More like

df1=df.stack().str.split(',').apply(pd.Series)
df1.stack().unstack(level=2).groupby(level=[0,1]).ffill().reset_index(level=[0,1])
Out[124]: 
   Name  Count test1 test2 test3
0  Emp1      1     X     A    a1
1  Emp1      1     Y     A    a2
0  Emp2      2     X     A    a3
1  Emp2      2     X     B    a3
2  Emp2      2     X     C    a3
0  Emp3      3     Z     C    a4
1  Emp3      3     Z     C    a5
2  Emp3      3     Z     C    a6

answered Sep 22 '18 at 02:47

BENY

317,841
20
164
234

can you please explain me the logic behind answer you posted, I have multiple columns where 3 columns need to split and join with other columns – PieSquare Sep 22 '18 at 02:59
1

@Mahesh first flatten the string column to single columns , then we just `unstack` – BENY Sep 22 '18 at 03:15

rafaelc · Answer 2 · 2018-09-22T03:35:22.513

(Not sure I understood right, but) Folllowing this answer, you can

expand(expand(df.drop('test3', 1), 'test1', ','), 'test2')

or

expand_all(df.drop('test3', axis=1), cols=['test1', 'test2'], seps=[',', ','])

where both output

    Name    test1   test2   Count
0   Emp1    X   A   1
1   Emp1    Y   A   1
2   Emp2    X   A   2
3   Emp2    X   B   2
4   Emp2    X   C   2
5   Emp3    Z   C   3

detail:

def expand(df, col, sep=','):
    r = df[col].str.split(sep)
    d = {c: df[c].values.repeat(r.str.len(), axis=0) for c in df.columns}
    d[col] = [i for sub in r for i in sub]
    return pd.DataFrame(d)

score 1 · Answer 3 · answered Sep 22 '18 at 04:25

1

I like using a comprehension

pd.DataFrame([
    (T.Name, T.Count, t1, t2)
    for T in df.itertuples()
    for t1, t2 in product(T.test1.split(','), T.test2.split(','))
], columns=['Name', 'Count', '0_x', '0_y'])

   Name  Count 0_x 0_y
0  Emp1      1   X   A
1  Emp1      1   Y   A
2  Emp2      2   X   A
3  Emp2      2   X   B
4  Emp2      2   X   C
5  Emp3      3   Z   C

answered Sep 22 '18 at 04:25

piRSquared

285,575
57
475
624

1

Nice answer ; ) – rafaelc Sep 22 '18 at 04:26

Join merge multiple dataframes

3 Answers3