Cycle through parallel lists deleting matches, until no more matches exist

Question

I have 3 parallel lists representing a 3-tuple (date, description, amount), and 3 new lists that I need to merge without creating duplicate entries. Yes, the lists have overlapping entries, however these duplicate entries are not grouped together (instead of all of the duplicates being 0 through x and all of the new entries being x through the end).

The problem I'm having is iterating the correct number of times to ensure all of the duplicates are caught. Instead, my code moves on with duplicates remaining.

for x in dates:
    MoveNext = 'false'
    while MoveNext == 'false':
        Reiterate = 'false'
        for a, b in enumerate(descriptions):
            if Reiterate == 'true':
                break
            if b in edescriptions:
                eindex = [c for c, d in enumerate(edescriptions) if d == b]
                for e, f in enumerate(eindex):
                    if Reiterate == 'true':
                        break
                    if edates[f] == dates[a]:
                        if eamounts[f] == amounts[a]:
                            del dates[a]
                            del edates[f]
                            del descriptions[a]
                            del edescriptions[f]
                            del amounts[a]
                            del eamounts[f]
                            Reiterate = 'true'
                            break
                        else:
                            MoveNext = 'true'
                    else:
                        MoveNext = 'true'
            else:
                MoveNext = 'true'

I don't know if it's a coincidence, but I'm currently getting exactly one half of the new items deleted and the other half remain. In reality, there should be far less than that remaining. That makes me think the for x in dates: is not iterating the correct number of times.

Answer 1

I suggest a different approach: Instead of trying to remove items from a list (or worse, several parallel lists), run through the input and yield only the data that passes your test --- in this case, data you haven't seen before. This is much easier with a single stream of input.

Your lists of data are crying out to be made into objects, since each piece (like the date) is meaningless without the other two... at least for your current purpose. Below, I start by combining each triplet into an instance of Record, a collections.namedtuple. They're great for this kind of use-once-and-throw-away work.

In the program below, build_records creates Record objects from your three input lists. dedup_records merges multiple streams of Record objects, using unique to filter out the duplicates. Keeping each function small (most of the main function is test data) makes each step easy to test.

#!/usr/bin/env python3

import collections
import itertools


Record = collections.namedtuple('Record', ['date', 'description', 'amount'])


def unique(records):
    '''
    Yields only the unique Records in the given iterable of Records.
    '''
    seen = set()
    for record in records:
        if record not in seen:
            seen.add(record)
            yield record
    return


def dedup_records(*record_iterables):
    '''
    Yields unique Records from multiple iterables of Records, preserving the
    order of first appearance.
    '''
    all_records = itertools.chain(*record_iterables)
    yield from unique(all_records)
    return


def build_records(dates, descriptions, amounts):
    '''
    Yields Record objects built from each date-description-amount triplet.
    '''
    for args in zip(dates, descriptions, amounts):
        yield Record(*args)
    return


def main():
    # Sample data
    dates_old = [
      '2000-01-01',
      '2001-01-01',
      '2002-01-01',
      '2003-01-01',
      '2000-01-01',
      '2001-01-01',
      '2002-01-01',
      '2003-01-01',
      ]
    dates_new = [
      '2000-01-01',
      '2001-01-01',
      '2002-01-01',
      '2003-01-01',
      '2003-01-01',
      '2002-01-01',
      '2001-01-01',
      '2000-01-01',
      ]
    descriptions_old = ['a', 'b', 'c', 'd', 'a', 'b', 'c', 'd']
    descriptions_new = ['b', 'b', 'c', 'a', 'a', 'c', 'd', 'd']
    amounts_old = [0, 1, 0, 1, 0, 1, 0, 1]
    amounts_new = [0, 0, 0, 0, 1, 1, 1, 1]
    old = [dates_old, descriptions_old, amounts_old]
    new = [dates_new, descriptions_new, amounts_new]

    for record in dedup_records(build_records(*old), build_records(*new)):
        print(record)
    return


if '__main__' == __name__:
    main()

This reduces the 16 input Records to 11:

Record(date='2000-01-01', description='a', amount=0)
Record(date='2001-01-01', description='b', amount=1)
Record(date='2002-01-01', description='c', amount=0)
Record(date='2003-01-01', description='d', amount=1)
Record(date='2000-01-01', description='b', amount=0)
Record(date='2001-01-01', description='b', amount=0)
Record(date='2003-01-01', description='a', amount=0)
Record(date='2003-01-01', description='a', amount=1)
Record(date='2002-01-01', description='c', amount=1)
Record(date='2001-01-01', description='d', amount=1)
Record(date='2000-01-01', description='d', amount=1)

Note that the yield from ... syntax requires Python 3.3 or greater.