Python: merging two lists where object "identity" isn't equality

Question

I'm trying to merge two lists, base and override, where base is supposed to be a larger list and override is a subset of the things in base. Where the elements overlap, I want the object in base to be overwritten by the one in override. The objects in each list are namedtuples with attributes al2000 and de2000 among them. Moreover I want to treat the objects as being "identical" when they have the same al2000 and de2000 values. What I have (which seems to work) is below, but this has nested loops, and I'm wondering if there is a better way to do this.

# Part of a function
final = []
for i in base:
    if all((i.al2000, i.de2000) != (k.al2000, k.de2000) for k in override):
        final.append(i)
    else:
        for k in override:
            if (i.al2000, i.de2000) == (k.al2000, k.de2000):
                final.append(k)
return final

In which case would you actually overwrite an item in base with an item from override? Seems you only want to append missing items. — Lasse V. Karlsen, Jul 22 '14 at 20:55

score 1 · Answer 1 · answered Jul 22 '14 at 21:04

You can use the for/else construct.

final = []
for i in base:
    for k in override:
        if (i.al2000, i.de2000) == (k.al2000, k.de2000):
            # found an override
            final.append(k)
            break
    else:
        final.append(i)

This solution still uses a nested for-loop, but it removes code duplication from your original solution (iterating over overrides, the comparison of i and k).

dirn · Answer 2 · 2014-07-22T21:14:01.007

1

You could use a dictionary and a couple of dictionary comprehensions.

uniques = {(x.al2000, x.de2000): x for x in base}
uniques.update({(x.al2000, x.de2000): x for x in override})
final = uniques.values()

Edit to retain original behavior about omitting extra values in override.

uniques = {(x.al2000, x.de2000): x for x in base}
for value in override:
    key = value.al2000, value.de2000
    if key in uniques:
        uniques[key] = value
# here's the comprehension version, although it's a bit rough on the eyes
# uniques.update({(x.al2000, x.de2000): x for x in override if (x.al2000, x.de2000) in uniques})
final = uniques.values()

edited Jul 22 '14 at 21:14

answered Jul 22 '14 at 21:04

dirn

19,454
5
69
74

This is not equivalent to original code because (1) it does not preserve order (2) elements in `override` which do not appear in `base` are included in `final`. otherwise, a nice elegant solution! – shx2 Jul 22 '14 at 21:07
I made an edit to address the second one. The first one cannot. Plus, depending on the size of `base`, maintaining a separate structure (i.e., `uniques`) could be memory intensive. – dirn Jul 22 '14 at 21:10

acushner · Answer 3 · 2014-07-23T12:51:35.607

0

since all you really want to do is look something up in overrides, make a dict out of it:

od = {(x.al2000, x.de2000): x for x in override}

now, just create the result:

res = [od.get((b.al2000, b.de2000), b) for b in base]

*not tested, but should be ok *edited for dirn's suggestion

edited Jul 23 '14 at 12:51

answered Jul 22 '14 at 22:05

acushner

9,595
1
34
34

1

You could also pass `b` to `od.get`. – dirn Jul 22 '14 at 22:20
nope, because b has more attributes than just `al2000` and `de2000` – acushner Jul 22 '14 at 22:26
1

I meant `od.get((b.al2000, b.de2000), b)`. – dirn Jul 22 '14 at 22:45

score 0 · Answer 4 · edited May 23 '17 at 10:33

This answers adopts @acushner's suggestion to use a dict, which might be the most natural approach in your case. To add to it, when dealing with overrides, ChainMap is useful and in many cases the most natural way to represent your data structure.

from collections import ChainMap
base_dct = {(x.al2000, x.de2000): x for x in base}
override_dct = {(x.al2000, x.de2000): x for x in override}
z = ChainMap(base_dct, override_dct)
# then access:
z.values()
# or:
z[(x.al2000, x.de2000)]

See this question for more info about ChainMaps.

Python: merging two lists where object "identity" isn't equality

4 Answers4