How to append dictionary from one column to anther column in pandas

Question

I have a dataframe like below:

df = pd.DataFrame({'id' : [1,2,3],
                  'attributes' : [{'dd' : True, 'budget' : '35k'}, {'dd' : True, 'budget' : '25k'}, {'dd' : True, 'budget' : '40k'}],
                  'prod.attributes' : [{'img' : 'img1.url', 'name' : 'millennials'}, {'img' : 'img2.url', 'name' : 'single'}, {'img' : 'img3.url', 'name' : 'married'}]})

df
    id  attributes                      prod.attributes
0   1   {'dd': True, 'budget': '35k'}   {'img': 'img1.url', 'name': 'millennials'}
1   2   {'dd': True, 'budget': '25k'}   {'img': 'img2.url', 'name': 'single'}
2   3   {'dd': True, 'budget': '40k'}   {'img': 'img3.url', 'name': 'married'}

I have multiple such columns wherein I need to append all columns that have attributes as suffix with the actual attributes column as below:

op = pd.DataFrame({'id' : [1,2,3],
              'attributes' : [{'dd' : True, 'budget' : '35k', 'prod' : {'img' : 'img1.url', 'name' : 'millennials'}}, \
                              {'dd' : True, 'budget' : '25k', 'prod' : {'img' : 'img2.url', 'name' : 'single'}}, 
                              {'dd' : True, 'budget' : '40', 'prod' : {'img' : 'img3.url', 'name' : 'married'}}]})

op

    id  attributes
0   1   {'dd': True, 'budget': '35k', 'prod': {'img': 'img1.url', 'name': 'millennials'}}
1   2   {'dd': True, 'budget': '25k', 'prod': {'img': 'img2.url', 'name': 'single'}}
2   3   {'dd': True, 'budget': '40', 'prod': {'img': 'img3.url', 'name': 'married'}}

I tried:

df['attributes'].apply(lambda x : x.update({'audience' : df['prod.attributes']}))

But I am getting all None. Could someone please help me on this.

mozway · Accepted Answer · 2022-11-29T08:52:13.733

2

More efficient than apply, use a loop and update the dictionaries in place:

for d1, d2 in zip(df['attributes'], df['prod.attributes']):
    d1['prod'] = d2

If you want to remove the original column use pop:

for d1, d2 in zip(df['attributes'], df.pop('prod.attributes')):
    d1['prod'] = d2

Updated dataframe:

   id                                                                         attributes
0   1  {'dd': True, 'budget': '35k', 'prod': {'img': 'img1.url', 'name': 'millennials'}}
1   2       {'dd': True, 'budget': '25k', 'prod': {'img': 'img2.url', 'name': 'single'}}
2   3      {'dd': True, 'budget': '40k', 'prod': {'img': 'img3.url', 'name': 'married'}}

timings

df = pd.concat([df]*10000, ignore_index=True)

%%timeit
for d1, d2 in zip(df['attributes'], df['prod.attributes']):
    d1['prod'] = d2
3.49 ms ± 137 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%%timeit
df['attributes'] = [{**a, **{'prod' : b}} 
                      for a, b in zip(df['attributes'], df['prod.attributes'])]
11.3 ms ± 384 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%%timeit
df.apply(lambda r: {**r['attributes'], **{'prod': r['prod.attributes']}}, axis=1)
173 ms ± 7.03 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

edited Nov 29 '22 at 08:52

answered Nov 29 '22 at 08:39

mozway

194,879
13
39
75

1

is `for loop` faster than `apply` in python in general? or is that just in this case it'd be faster – Karthik S Nov 29 '22 at 08:46
3

Most of the time yes, a list comprehension/loop is faster than **pandas `apply` on rows**. Also, if you want to update in place, `apply` is not well suited as it would return a Series of `None`. – mozway Nov 29 '22 at 08:47
2

dictionary unpacking (`**`) is also quite slow – mozway Nov 29 '22 at 08:48
1

See the comparison in the update – mozway Nov 29 '22 at 08:52
Could you please let me know why `apply` is returning a `Series` of `None`. I actually got that as output – Karthik S Nov 29 '22 at 09:08
1

Because `update` is in place and returns nothing. This is the same as doing `[print(i) for i in range(3)]` in python, it's an inappropriate use of a list comprehension. – mozway Nov 29 '22 at 09:14

jezrael · Answer 2 · 2022-11-29T08:42:24.727

1

Use ** for merge both dictionaries in list comprehension, DataFrame.pop is used for remove column after using:

df['attributes'] = [{**a, **{'prod' : b}} 
                      for a, b in zip(df['attributes'], df.pop('prod.attributes'))]
print (df)
   id                                         attributes
0   1  {'dd': True, 'budget': '35k', 'prod': {'img': ...
1   2  {'dd': True, 'budget': '25k', 'prod': {'img': ...
2   3  {'dd': True, 'budget': '40k', 'prod': {'img': ...

edited Nov 29 '22 at 08:42

answered Nov 29 '22 at 08:40

jezrael

822,522
95
1,334
1,252

Could you please let me know what `**` are for? – Karthik S Nov 29 '22 at 08:41

How to append dictionary from one column to anther column in pandas

2 Answers2

timings