Creating new column with sets in Pandas DataFrame

Question

Hello_world!

I have a DataFrame like this:

from pandas import DataFrame

df = DataFrame({"A": ['sd', 'df', 'gh', 'rv'],
                "B": ['hj', '4r', 'tg', '2s'],
                "C": ['hf', 'qw', 'e4', '7u'],
                "D": ['1q', 'nc', 'xf', '7y'],
                "E": ['9i', 'g7', 'ce', 'x3']})

or

    A   B   C   D   E
0   sd  hj  hf  1q  9i
1   df  4r  qw  nc  g7
2   gh  tg  e4  xf  ce
3   rv  2s  7u  7y  x3

I need to create a new column that will contain values of the set type, consisting of the values of the first five columns.

Expected result is:

    A   B   C   D   E   F
0   sd  hj  hf  1q  9i  {'sd','hj', 'hf', '1q', '9i'}
1   df  4r  qw  nc  g7  {'df','4r', 'qw', 'nc', 'g7'}
2   gh  tg  e4  xf  ce  {'gh','tg', 'e4', 'xf', 'ce'}
3   rv  2s  7u  7y  x3  {'rv','2s', '7u', '7y', 'x3'}

print(type(df.loc[0, 'F']))    # <class 'set'>
print(type(df.loc[0, 'A']))    # <class 'str'>

My code:

from pandas import DataFrame

df = DataFrame({"A": ['sd', 'df', 'gh', 'rv'],
                "B": ['hj', '4r', 'tg', '2s'],
                "C": ['hf', 'qw', 'e4', '7u'],
                "D": ['1q', 'nc', 'xf', '7y'],
                "E": ['9i', 'g7', 'ce', 'x3']})

f = {df.loc[0, 'A'], df.loc[0, 'B'], df.loc[0, 'C'], df.loc[0, 'D'], df.loc[0, 'E']}

df = df.assign(F = f)

print(df)

...have ValueError: Length of values (5) does not match length of index (4).

If I rewrite the code so that the length of the values matches the length of the index:

from pandas import DataFrame

df = DataFrame({"A": ['sd', 'df', 'gh', 'rv'],
                "B": ['hj', '4r', 'tg', '2s'],
                "C": ['hf', 'qw', 'e4', '7u'],
                "D": ['1q', 'nc', 'xf', '7y'],
                "E": ['9i', 'g7', 'ce', 'x3']})

f = {df.loc[0, 'A'], df.loc[0, 'B'], df.loc[0, 'C'], df.loc[0, 'D']}

df = df.assign(F = f)

print(df)

...I have TypeError: 'set' type is unordered.

I ask the respected community for help to solve my problem.

try `df['F'] = df.apply(set, axis=1)`, what you are trying to do is not quite correct, here you are creating a set of all the data not row-wise that is why it is complaining about length mismatch, you need to create set row-wise. — Epsi95, Feb 11 '22 at 05:23

score 4 · Answer 1 · answered Feb 11 '22 at 05:25

Simply use:

df['F'] = df.apply(set, axis=1)

Note however that you have no control over the displayed order of sets as they are unordered containers.

Output:

    A   B   C   D   E                     F
0  sd  hj  hf  1q  9i  {hj, 1q, 9i, sd, hf}
1  df  4r  qw  nc  g7  {nc, df, qw, g7, 4r}
2  gh  tg  e4  xf  ce  {e4, xf, gh, ce, tg}
3  rv  2s  7u  7y  x3  {7u, x3, rv, 2s, 7y}

Mayank Porwal · Accepted Answer · 2022-02-11T17:30:24.957

2

Use List comprehension for better performance:

In [1069]: df['F'] = [set(i) for i in df.values]

In [1070]: df
Out[1070]: 
    A   B   C   D   E                     F
0  sd  hj  hf  1q  9i  {sd, 1q, hf, 9i, hj}
1  df  4r  qw  nc  g7  {g7, 4r, nc, df, qw}
2  gh  tg  e4  xf  ce  {xf, gh, ce, e4, tg}
3  rv  2s  7u  7y  x3  {2s, x3, 7y, rv, 7u}

OR as suggested by @jezrael:

df['F'] = [set(i) for i in df.to_numpy()]

To use specific columns:

If you want to use let's say : B,C,D, do this:

In [1103]: df['F'] = [set(i) for i in df[['B', 'C', 'D']].to_numpy()]

In [1104]: df
Out[1104]: 
    A   B   C   D   E             F
0  sd  hj  hf  1q  9i  {hf, hj, 1q}
1  df  4r  qw  nc  g7  {nc, qw, 4r}
2  gh  tg  e4  xf  ce  {xf, tg, e4}
3  rv  2s  7u  7y  x3  {7y, 7u, 2s}

Performance timings:

@mozway's solution:

In [1078]: %timeit df.apply(set, axis=1)
395 µs ± 24.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

My solution:

In [1079]: %timeit [set(i) for i in df.values]
7.3 µs ± 31.8 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

edited Feb 11 '22 at 17:30

answered Feb 11 '22 at 05:27

Mayank Porwal

33,470
8
37
58

For better performance than what? – ddejohn Feb 11 '22 at 06:12
@ddejohn Than `df.apply`. – Mayank Porwal Feb 11 '22 at 06:49
I find that surprising. Mind including some benchmarks? – ddejohn Feb 11 '22 at 06:50
@ddejohn I've added some. Please have a look. – Mayank Porwal Feb 11 '22 at 07:05
1

https://stackoverflow.com/a/54508052/2901002 – jezrael Feb 11 '22 at 07:09
1

Your solution looks like the best. But how to change the code to use only specific columns, for example, С and E or B, C, D etc. – Alexander Grenkov Feb 11 '22 at 10:36
Interesting. I'd be curious to see how the performance difference scales up. – ddejohn Feb 11 '22 at 16:25

Creating new column with sets in Pandas DataFrame

2 Answers2