How do I perform One Hot Encoding on lists in a pandas column?

Question

Suppose I have a dataframe of which one column is a list (of a unknown values and length) for example:

df = pd.DataFrame(
 {'messageLabels': [['Good', 'Other', 'Bad'],['Bad','Terrible']]}
)

I came across this solution but it isnt what I am looking for. How best to extract a Pandas column containing lists or tuples into multiple columns

in theory the resulting df would look like

messageLabels             | Good| Other| Bad| Terrible
--------------------------------------------------------
['Good', 'Other', 'Bad']  | True| True |True| False
--------------------------------------------------------
['Bad','Terrible']        |False|False |True| True

See above

Answer 1

Succint

df.join(df.messageLabels.str.join('|').str.get_dummies().astype(bool))

        messageLabels   Bad   Good  Other  Terrible
0  [Good, Other, Bad]  True   True   True     False
1     [Bad, Terrible]  True  False  False      True

---

sklearn

from sklearn.preprocessing import MultiLabelBinarizer

mlb = MultiLabelBinarizer()
dum = mlb.fit_transform(df.messageLabels)

df.join(pd.DataFrame(dum.astype(bool), df.index, mlb.classes_))

        messageLabels   Bad   Good  Other  Terrible
0  [Good, Other, Bad]  True   True   True     False
1     [Bad, Terrible]  True  False  False      True

---

Overdone

n = len(df)
i = np.arange(n)
l = [*map(len, df.messageLabels)]
j, u = pd.factorize(np.concatenate(df.messageLabels))

o = np.zeros((n, len(u)), bool)
o[i.repeat(l), j] = True

df.join(pd.DataFrame(o, df.index, u))

        messageLabels   Good  Other   Bad  Terrible
0  [Good, Other, Bad]   True   True  True     False
1     [Bad, Terrible]  False  False  True      True

---

Messing around

And inspire by Andy

df.join(pd.DataFrame([dict.fromkeys(x, True) for x in df.messageLabels]).fillna(False))

        messageLabels   Bad   Good  Other  Terrible
0  [Good, Other, Bad]  True   True   True     False
1     [Bad, Terrible]  True  False  False      True

Answer 2

Another way is to use the apply and the Series constructor:

In [11]: pd.get_dummies(df.messageLabels.apply(lambda x: pd.Series(1, x)) == 1)
Out[11]:
    Good  Other   Bad  Terrible
0   True   True  True     False
1  False  False  True      True

where

In [12]: df.messageLabels.apply(lambda x: pd.Series(1, x))
Out[12]:
   Good  Other  Bad  Terrible
0   1.0    1.0  1.0       NaN
1   NaN    NaN  1.0       1.0

---

To get your desired output:

In [21]: res = pd.get_dummies(df.messageLabels.apply(lambda x: pd.Series(1, x)) == 1)

In [22]: df[res.columns] = res

In [23]: df
Out[23]:
        messageLabels   Good  Other   Bad  Terrible
0  [Good, Other, Bad]   True   True  True     False
1     [Bad, Terrible]  False  False  True      True

Answer 3

I would do this using get_dummies and sum (or max, either of them work):

tmp = pd.DataFrame(df['messageLabels'].tolist())
pd.get_dummies(tmp, prefix='', prefix_sep='').max(level=0, axis=1).astype(bool)

    Bad   Good  Other  Terrible
0  True   True   True     False
1  True  False  False      True

You can combine this with df using join:

df.join(pd.get_dummies(tmp, prefix='', prefix_sep='')
          .max(level=0, axis=1)
          .astype(bool))

        messageLabels   Bad   Good  Other  Terrible
0  [Good, Other, Bad]  True   True   True     False
1     [Bad, Terrible]  True  False  False      True

---

You can also stack and pivot_table:

(pd.DataFrame(df['messageLabels'].tolist())
   .stack()
   .reset_index()
   .pivot_table(index='level_0', columns=0, aggfunc='size', fill_value=0)
   .astype(bool))

0         Bad   Good  Other  Terrible
level_0                              
0        True   True   True     False
1        True  False  False      True