Python: Remove duplicate lists in list of lists

Question

Given list that looks like:

list = [["A"], ["B"], ["A","B"], ["B","A"], ["A","B","C"], ["B", "A", "C"]]

How do I return

final_list = [["A"], ["B"], ["A", "B"], ["A", "B", "C"]]

Note that I treat ["A","B"] to be same as ["B","A"] and ["A","B","C"] same as ["B", "A", "C"].

Answer 1

Try this :

list_ = [["A"], ["B"], ["A","B"], ["B","A"], ["A","B","C"], ["B", "A", "C"]]
l = list(map(list, set(map(tuple, map(set, list_)))))

Output :

[['A', 'B'], ['B'], ['A', 'B', 'C'], ['A']]

This process goes through like :

1. First convert each sub-list into a set. Thus ['A', 'B'] and ['B', 'A'] both are converted to {'A', 'B'}.
2. Now convert each of them to a tuple for removing duplicate items as set() operation can not be done with set sub-items in the list.
3. With set() operation make a list of unique tuples.
4. Now convert each tuple items in the list into list type.

This is equivalent to :

list_ = [['A'], ['B'], ['A', 'B'], ['B', 'A'], ['A', 'B', 'C'], ['B', 'A', 'C']]
l0 = [set(i) for i in list_]
# l0 = [{'A'}, {'B'}, {'A', 'B'}, {'A', 'B'}, {'A', 'B', 'C'}, {'A', 'B', 'C'}]
l1 = [tuple(i) for i in l0]
# l1 = [('A',), ('B',), ('A', 'B'), ('A', 'B'), ('A', 'B', 'C'), ('A', 'B', 'C')]
l2 = set(l1)
# l2 = {('A', 'B'), ('A',), ('B',), ('A', 'B', 'C')}
l = [list(i) for i in l2]
# l = [['A', 'B'], ['A'], ['B'], ['A', 'B', 'C']]

Answer 2

One possible solution:

lst = [["A"], ["B"], ["A","B"], ["B","A"], ["A","B","C"], ["B", "A", "C"]]

print([
    list(i) 
    for i in sorted(
        set(
            tuple(sorted(i)) 
            for i in lst
        ), 
        key=lambda k: (len(k), k)
    )
])

Prints:

[['A'], ['B'], ['A', 'B'], ['A', 'B', 'C']]

Answer 3

When the data you want to handle has to be both unique and unordered, a better choice of data structure are set and frozenset.

A set is an unordered container of unique values.

A frozenset is a set which cannot be mutated, it is thus hashable which allows it to be contained into another set.

Example

lst = [["A"], ["B"], ["A","B"], ["B","A"], ["A","B","C"], ["B", "A", "C"]]    

data = {frozenset(el) for el in lst}

print(data)

Output

{frozenset({'B'}), frozenset({'A', 'B'}), frozenset({'A', 'C', 'B'}), frozenset({'A'})}

Answer 4

l = [["A"], ["B"], ["A","B"], ["B","A"], ["A","B","C"], ["B", "A", "C"]]
[list(i) for i in {tuple(sorted(i)) for i in l}]

Answer 5

The following is a equality partition. It works on any list of any type that has equality defined for it. This is worse than a hash partition as it is quadratic time.

def partition(L, key=None):

    if key is None:
        key = lambda x: x

    parts = []
    for item in L:
        for part in parts:
            if key(item) == key(part[0]):
               part.append(item)
               break
        else:
            parts.append([item])
    return parts

def unique(L, key=None):
    return [p[0] for p in partition(L, key=key)]

alist = [["A"], ["B"], ["A","B"], ["B","A"], ["A","B","C"], ["B", "A", "C"]]

unique(alist)
# results in [['A'], ['B'], ['A', 'B'], ['B', 'A'], ['A', 'B', 'C'], ['B', 'A', 'C']]

unique(alist, key=lambda v: tuple(sorted(v)))
# results in [['A'], ['B'], ['A', 'B'], ['A', 'B', 'C']]