Getting the difference between 2 lists that contain dictionaries

Question

list1 = [{'key1': 'item1'}, {'key2': 'item2'}]
list2 = [{'key1': 'item1'}, {'key2': 'item2'}, {'key3': 'item3'}]

Is there a way to get the difference between those two lists?

Basically, I need a scaleable way to get the differences between 2 lists that contain dictionaries. So I'm trying to compare those lists, and just get a return of {'key3': 'item3'}

^ not necessarily a duplicate since the elements here are dictionaries which are not readily hashable using a set — LeKhan9, Nov 11 '18 at 00:58
Do you really only have dictionaries with one key/value, or is that just for demonstration? Because it would probably make a lot more sense to just have one dictionary `{key1: item1, key2: item2, ...}` rather than a list. — Alex Hall, Nov 11 '18 at 09:40
"Is there a way to get the difference between those two lists?" – You do it by writing a program which does that. If you have a problem with your program, carefully read the documentation of all the methods, classes, modules, and libraries you are using, write tests for your programs, trace the execution with pen and paper, single-step it in a debugger, then sleep on it, start again from the beginning, sleep on it again, and *then and only then* narrow your problem down to a concise, focused, simple, short, reproducible [mcve] and ask a specific, focused, narrow question on [so]. — Jörg W Mittag, Nov 11 '18 at 09:54

score 13 · Accepted Answer · answered Nov 11 '18 at 00:56

13

You could use a list comprehension:

list1 = [{'key1': 'item1'}, {'key2': 'item2'}]
list2 = [{'key1': 'item1'}, {'key2': 'item2'}, {'key3': 'item3'}]

print([x for x in list2 if x not in list1])

Which will give [{'key3': 'item3'}]

answered Nov 11 '18 at 00:56

Simon

9,762
15
62
119

score 3 · Answer 2 · answered Nov 11 '18 at 01:13

You can use set() with a comprehension like this example:

def get_diff(elm1, elm2):
    a = set((m, n) for k in elm1 for m, n in k.items())
    b = set((m, n) for k in elm2 for m, n in k.items())
    if len(b) > len(a):
        return dict(b - a)
    return dict(a - b)


list1 = [{'key1': 'item1'}, {'key2': 'item2'}]
list2 = [{'key1': 'item1'}, {'key2': 'item2'}, {'key3': 'item3'}]
get_diff(list1, list2)

Output:

{'key3': 'item3'}

score 1 · Answer 3 · answered Nov 11 '18 at 00:56

1

in_list1_not_in_list2 = [i for i in list1 if i not in list2]
in_list2_not_in_list1 = [i for i in list2 if i not in list1]

answered Nov 11 '18 at 00:56

qwerty asdf

25
6

LeKhan9 · Answer 4 · 2018-11-11T01:40:57.027

Since dictionaries are not hashable there isn't an easy way to hash, but since we have one key and one val per dictionary, we can build our own key! So you can do something like this:

list1_set = set()

for dictionary in list1:
    key = dictionary.keys()[0]
    vals = dictionary.values()[0]
    custom_key = '{}|{}'.format(key,vals)
    list1_set.add(custom_key)

differences = []
for dictionary in list2:
    key = dictionary.keys()[0]
    vals = dictionary.values()[0]
    custom_key = '{}|{}'.format(key,vals)

    if custom_key not in list1_set:
        differences.append(dictionary)

print differences

output:

[{'key3': 'item3'}]

Not this solution is much more scalable then simply iterating through the first list because of the constant lookup ability.

Rich Tier · Answer 5 · 2018-11-12T08:13:27.973

You can inform the dictionary how to hash itself and then you could use sets

import json

class HashableDict(dict):   
    def __hash__(self):
        # convert the dictionary to something hashable - in this case a str
        return hash(json.dumps(self))

then you can do

hashable_list1 = map(HashableDict, list1)
hashable_list2 = map(HashableDict, list2)
set(hashable_list2).difference(hashable_list1)

difference gives you the elements in lists2 that are not in list1.

If you wanted all the difference, so all the items that are not in both lists, do:

set(hashable_list2).symmetric_difference(hashable_list1)

Note this will not work for all dictionaries (e.g., dictionaries containing objects the json.dumps cannot work with) unless you handle those explicitly too with a custom JSONEncoder

RoadRunner · Answer 6 · 2018-11-11T02:02:33.423

You can also try using set.symmetric_difference() to get the difference between the sets both ways:

list1 = [{'key1': 'item1'}, {'key2': 'item2'}]
list2 = [{'key1': 'item1'}, {'key2': 'item2'}, {'key3': 'item3'}]

set1 = set(tuple(x.items())[0] for x in list1)
set2 = set(tuple(x.items())[0] for x in list2)

print([dict(list(set1.symmetric_difference(set2)))])
# [{'key3': 'item3'}]

print([dict(list(set2.symmetric_difference(set1)))])
# [{'key3': 'item3'}]

Another way would be to use itertools.filterfalse():

from itertools import filterfalse

diff1 = list(filterfalse(lambda d: d in list2, list1))
diff2 = list(filterfalse(lambda d: d in list1, list2))

print(diff1 + diff2)
# [{'key3': 'item3'}]

Getting the difference between 2 lists that contain dictionaries

6 Answers6