Remove duplicate 2D dict values

Question

I have the following list of dicts:

defaultdict(<class 'list'>,
            {'192.168.20.10/32': [{1: 'aaaaa11111\n'},
                                  {2: 'bbbbb11111\n'},
                                  {2: 'bbbbb11111\n'},
                                  {2: 'bbbbb11111\n'},
                                  {2: 'bbbbb11111\n'}],
             '192.168.20.20/32': [{1: 'aaaaa11111\n'},
                                  {2: 'aaaaa11111\n'},
                                  {2: 'bbbbb11111\n'},
                                  {2: 'bbbbb11111\n'},
                                  {2: 'bbbbb11111\n'},
                                  {2: 'bbbbb11111\n'}]})

I would like to iterate over this list object and remove the duplicate list items.

The end result would look something like this:

defaultdict(<class 'list'>,
                {'192.168.20.10/32': [{1: 'aaaaa11111\n'},
                                      {2: 'bbbbb11111\n'}],
                 '192.168.20.20/32': [{1: 'aaaaa11111\n'},
                                      {2: 'bbbbb11111\n'}]})

Please note that I am using defaultdict from collections above:

from collections import defaultdict

It might be more worthwhile to correct it at the source, i.e don't create this pointless list of 1-element dictionaries in the first place, and contain a dictionary directly. — matszwecja, Aug 10 '22 at 13:48
Shouldn't be '192.168.20.20/32': [{1: 'aaaaa11111\n'}, {2: 'bbbbb11111\n'}, {2: 'aaaaa11111\n'}]}? — funnydman, Aug 10 '22 at 13:54

score 2 · Accepted Answer · answered Aug 10 '22 at 14:08

Maybe something like this (if resulting order does not matter):

adict = {
    '192.168.20.10/32': [
        {1: 'aaaaa11111\n'},
        {2: 'bbbbb11111\n'},
        {2: 'bbbbb11111\n'},
        {2: 'bbbbb11111\n'},
        {2: 'bbbbb11111\n'}
    ],
    '192.168.20.20/32': [
        {1: 'aaaaa11111\n'},
        {2: 'aaaaa11111\n'},
        {2: 'bbbbb11111\n'},
        {2: 'bbbbb11111\n'},
        {2: 'bbbbb11111\n'},
        {2: 'bbbbb11111\n'}
    ]
}

for k, v in adict.items():
    adict[k] = list(map(dict, (set([tuple(obj.items()) for obj in v]))))
print(adict)

11574713 · Answer 2 · 2022-08-10T23:52:39.357

I've used the approach from https://stackoverflow.com/a/11092607/11574713.

Calling your dictionary first_dict, run the dicts through a set by converting them into a string (JSON), and then bring them back to dicts.

import json

for k in first_dict:
    first_dict[k] = [json.loads(j) for j in set([json.dumps(i) for i in first_dict[k]])]

defaultdict(list,
            {'192.168.20.10/32': [{'2': 'bbbbb11111\n'},
              {'1': 'aaaaa11111\n'}],
             '192.168.20.20/32': [{'2': 'aaaaa11111\n'},
              {'2': 'bbbbb11111\n'},
              {'1': 'aaaaa11111\n'}]})

Remove duplicate 2D dict values

2 Answers2