Pandas: split list in column into multiple rows

Question

I have a question regarding splitting a list in a dataframe column into multiple rows.

Let's say I have this dataframe:

  Job position   Job type  id
0          [6]        [1]   3
1       [2, 6]  [3, 6, 5]   4
2          [1]        [9]  43

I would like every single combination of numbers, so the final result would be:

   id    Job position  Job type
0   3         6.0       1.0
1   4         2.0       3.0
2   4         2.0       6.0
3   4         2.0       5.0
4   4         6.0       3.0
5   4         6.0       6.0
6   4         6.0       5.0
7  43         1.0       9.0

Because right now I get this result:

   id    Job position  Job type
0   3         6.0       1.0
1   4         2.0       3.0
2   4         6.0       6.0
3   4         NaN       5.0
4  43         1.0       9.0

In order to get the result above, I did:

df = df.set_index(['id'])
(df.apply(lambda x: pd.DataFrame(x.tolist(),index=x.index)
                        .stack()
                        .rename(x.name)).reset_index())

score 18 · Accepted Answer · answered May 07 '18 at 16:01

Similar to Scott Boston's suggestion, I suggest you explode the columns separately, then merge them together.

For example, for 'Job position':

>>> df['Job position'].apply(pd.Series).reset_index().melt(id_vars='index').dropna()[['index', 'value']].set_index('index')
    value
index   
0   6.0
1   2.0
2   1.0
1   6.0

And, all together:

df = pd.DataFrame({'Job position': [[6], [2, 6], [1]], 'Job type': [[1], [3, 6, 5], [9]], 'id': [3, 4, 43]})
jobs = df['Job position'].apply(pd.Series).reset_index().melt(id_vars='index').dropna()[['index', 'value']].set_index('index')
types = df['Job type'].apply(pd.Series).reset_index().melt(id_vars='index').dropna()[['index', 'value']].set_index('index')
>>> pd.merge(
    pd.merge(
        jobs,
        types,
        left_index=True,
        right_index=True),
    df[['id']],
    left_index=True,
    right_index=True).rename(columns={'value_x': 'Job positions', 'value_y': 'Job type'})
Job positions   Job type    id
0   6.0 1.0 3
1   2.0 3.0 4
1   2.0 6.0 4
1   2.0 5.0 4
1   6.0 3.0 4
1   6.0 6.0 4
1   6.0 5.0 4
2   1.0 9.0 43

piRSquared · Answer 2 · 2018-05-07T19:26:52.620

Use a comprehension

pd.DataFrame([
    [p, t, i] for P, T, i in df.values
    for p in P for t in T
], columns=df.columns)

   Job position  Job type  id
0             6         1   3
1             2         3   4
2             2         6   4
3             2         5   4
4             6         3   4
5             6         6   4
6             6         5   4
7             1         9  43

Alternatives to iterating over values

pd.DataFrame([
    [p, t, i] for P, T, i in df.itertuples(index=False)
    for p in P for t in T
], columns=df.columns)

z = zip(df['Job position'], df['Job type'], df['id'])
pd.DataFrame([
    [p, t, i] for P, T, i in z
    for p in P for t in T
], columns=df.columns)

To generalize this solution to accommodate any number of columns

pd.DataFrame([
    [p, t] + a for P, T, *a in df.values
    for p in P for t in T
], columns=df.columns)

   Job position  Job type  id
0             6         1   3
1             2         3   4
2             2         6   4
3             2         5   4
4             6         3   4
5             6         6   4
6             6         5   4
7             1         9  43

Thanks for your answer piRSquared, it works great, however I want to merge even more columns like this, then your approach might be a bit tricky :) — Mathias Lund, May 07 '18 at 19:13
There is a way to adapt it. But no worries. I'll update with an example if I get a minute. — piRSquared, May 07 '18 at 19:14

score 4 · Answer 3 · answered May 07 '18 at 16:03

From data frame constructor

s1=df.Jobposition.str.len()

s2=df.Jobtype.str.len()
pd.DataFrame({'id':df.id.repeat(s1*s2),
  'Jobposition':np.concatenate([np.repeat(x,y) for x,y in zip(df.Jobposition,s2)]),
  'Jobtype':np.concatenate(np.repeat(df.Jobtype,s1).values)})

   Jobposition  Jobtype  id
0            6        1   3
1            2        3   4
1            2        6   4
1            2        5   4
1            6        3   4
1            6        6   4
1            6        5   4
2            1        9  43

score 0 · Answer 4 · answered May 12 '18 at 15:29

import itertools
dfres = pd.DataFrame([j+(i[2],) for i in df.values for j in itertools.product(*i[0:2])]
        ,columns=df.columns)

   Job position  Job type  id
0             6         1   3
1             2         3   4
2             2         6   4
3             2         5   4
4             6         3   4
5             6         6   4
6             6         5   4
7             1         9  43

Pandas: split list in column into multiple rows

4 Answers4

Linked

Related