Finding differences of two list of dictionaries in Python

Question

I have two dictionaries like below:

prev = [ 
  { 'id': 0, 'name': 'a' }, 
  { 'id': 1, 'name': 'b' }, 
  { 'id': 2, 'name': 'c' } 
]

current = [ 
  { 'id': 1, 'name': 'b' }, 
  { 'id': 2, 'name': 'c' }, 
  { 'id': 3, 'name': 'e' }, 
  { 'id': 4, 'name': 'f' } 
]

I want to get the difference of them, the result should be like below:

result = [ 
  { 'id': 3, 'name': 'e' }, 
  { 'id': 4, 'name': 'f' } 
]

Only the difference of those two should appear in result list, mu solution is like below

common = []
for c in current:
  for p in prev:
    if c['name'] == p['name']:
      common.append(c)

print(common)

I'm trying to find the common items between two and then subtract it from current list but I don't know hot to handle it. If I am using the wrong procedure to resolve the issue, is there another way I can find these two diffs?

I tried to search a lot but all the results I found were just comparing two list of integers, which in my case is list of dictionaries.

Also note that the id key is just for separating those items, let's compare by name, let's consider I want to remove commons from current and remain the rest just in current list. By that I mean that I don't need name: a and name: b from prev list.

What about `{ 'id': 0, 'name': 'a' }`? It's in `prev` but not `current`. — wjandrea, Dec 26 '19 at 15:02
i dont need any prev value from `prev` list , i only want new items that comes in `current` list — amdev, Dec 26 '19 at 15:04
also note that in real world i have a link key in both dicts to compare if i could find the not equal links in `current` list then i found the desired items — amdev, Dec 26 '19 at 15:07
What is with duplicate names? Can a `name` not appear under different id's? — Darkonaut, Dec 26 '19 at 15:49

wjandrea · Accepted Answer · 2019-12-26T18:13:03.397

Simple

From the data you posted, you can compare the whole dicts, so just find dicts in current that are not in prev:

new = [d for d in current if d not in prev]
print(new)  # -> [{'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]

Complex

If your real-world data might have differing ids, the solution needs to get more complex.

Since only the names are important, make a set of common names. Then you can loop over the dicts and check whether the name is in the common set.

prev = [{'id': 0, 'name': 'a'}, {'id': 1, 'name': 'b'}, {'id': 2, 'name': 'c'}]
current = [{'id': 1, 'name': 'b'}, {'id': 2, 'name': 'c'}, {'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]

prev_names, current_names = [{d['name'] for d in x} for x in (prev, current)]  # [{'c', 'b', 'a'}, {'c', 'b', 'f', 'e'}]
common_names = prev_names & current_names  # {'b', 'c'}
new = [d for d in current if d['name'] not in common_names]
print(new)  # -> [{'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]

This is also easy to adapt to getting names in prev that are not common:

old = [d for d in prev if d['name'] not in common_names]
print(old)  # -> [{'id': 0, 'name': 'a'}]

Henry Harutyunyan · Answer 2 · 2019-12-26T15:20:50.287

3

This will do the job

prev = [ { 'id': 0, 'name': 'a' }, { 'id': 1, 'name': 'b' }, { 'id': 2, 'name': 'c' } ]
current = [ { 'id': 1, 'name': 'b' }, { 'id': 2, 'name': 'c' }, { 'id': 3, 'name': 'e' }, { 'id': 4, 'name': 'f' } ]

common = []
for c in current:
    if not any(c['id'] == p['id'] and c['name'] == p['name'] for p in prev):
        common.append(c)

print(common)

Return True if any element of the iterable is true. If the iterable is empty, return False

Alse, as @wjandrea noted in the comments, this

new = [c for c in current if c not in prev]

is also a fair and nice answer. But note that it only works if comparing the whole dicts

edited Dec 26 '19 at 15:20

answered Dec 26 '19 at 15:12

Henry Harutyunyan

2,355
1
16
22

1

Based on OP's code I think the `'id'` actually doesn't matter, but if it does you might as well compare the whole dicts, `any(c == p for p in prev)` – wjandrea Dec 26 '19 at 15:13
`common` is the wrong name, maybe `new` would be better? – wjandrea Dec 26 '19 at 15:15
@wjandrea yeah right. One can use `c == p` if the whole diet is compared – Henry Harutyunyan Dec 26 '19 at 15:15
@wjandrea used the names in the OP – Henry Harutyunyan Dec 26 '19 at 15:15
I realized you used the names in the OP, but @a_m_dev was trying to create a list of common names first – wjandrea Dec 26 '19 at 15:16
Here's a one-line version: `new = [c for c in current if not any(c == p for p in prev)]` **Edit** oh wait, you can simplify this even further: `new = [c for c in current if c not in prev]` – wjandrea Dec 26 '19 at 15:17

score 3 · Answer 3 · answered Dec 26 '19 at 15:22

3

If I understood correctly, you want only the items that appear in current and did not appear in prev.

Something like this should work

 prev_names = set(map(lambda x: x['name'], prev))
 new_items = [item for item in current if item['name'] not in prev_names]
 new_items # [{'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]

answered Dec 26 '19 at 15:22

RodMendez

51
4

4

what's the point of using `map` with `lambda` if you are just converting back to a `set`. would be more pythonic to do `set(item['name'] for item in prev)` – gold_cy Dec 26 '19 at 15:24
1

I think that difference is a matter of taste, I would say both are pythonic, but is a valid comment – RodMendez Dec 26 '19 at 15:26
2

actually I disagree, comprehensions are more readable especially in this context, not to mention there is no point of using `map` considering that you convert it into a set anyways – gold_cy Dec 26 '19 at 15:27
1

map is not creating a dict, is just creating an iterator, same as the for, so I don't see the difference. I agree with this "comprehensions are more readable especially in this context", that's why I upvoted ur comment – RodMendez Dec 26 '19 at 15:34
where did I say `map` is creating a dictionary? nowhere.... – gold_cy Dec 26 '19 at 15:36

score 2 · Answer 4 · answered Dec 26 '19 at 15:29

2

Code

import itertools
list(itertools.filterfalse(lambda x: x in prev, current))

Output:

[{'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]

answered Dec 26 '19 at 15:29

Ekaba Bisong

2,918
2
23
38

1

Simpler: `[d for d in current if d not in prev]` – wjandrea Dec 26 '19 at 15:33
@wjandrea :D :D – Ekaba Bisong Dec 26 '19 at 15:33
2

@wjandrea despite simplicity, `itertools` is usually faster as it is implemeted in C. – Henry Harutyunyan Dec 26 '19 at 15:33
Not is this particular case though :D – Henry Harutyunyan Dec 26 '19 at 15:34

score 2 · Answer 5 · answered Dec 26 '19 at 15:40

Based on all of the answers here is a little benchmark

import timeit
import itertools

prev = [{'id': 0, 'name': 'a'}, {'id': 1, 'name': 'b'}, {'id': 2, 'name': 'c'}]
current = [{'id': 1, 'name': 'b'}, {'id': 2, 'name': 'c'}, {'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]


def f1():
    prev_names = set(map(lambda x: x['name'], prev))
    new_items = [item for item in current if item['name'] not in prev_names]
    return new_items


def f2():
    common = []
    for c in current:
        if not any(c['id'] == p['id'] and c['name'] == p['name'] for p in prev):
            common.append(c)

    return common


def f3():
    return list(itertools.filterfalse(lambda x: x in prev, current))


print(f1())
print(timeit.timeit("f1()", setup="from __main__ import f1"))
print(f2())
print(timeit.timeit("f2()", setup="from __main__ import f2"))
print(f3())
print(timeit.timeit("f3()", setup="from __main__ import f3"))

[{'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]
0.8235890520736575
[{'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]
2.0767332719406113
[{'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]
0.864271447993815

Finding differences of two list of dictionaries in Python

5 Answers5

Simple

Complex