How to remove duplicates in a list containing lists of sets?

Question

I am trying to remove the duplicates in the following list. itertools did not work here 'cause the internal lists are set instead of integers.

[[{1, 2, 3, 8, 9, 10}, {4, 5, 7}, {5, 6, 7}], 
 [{1, 2, 3, 4, 5}, {6, 7, 8, 9, 10}], 
 [{4, 5, 7}, {1, 2, 3, 8, 9, 10}, {5, 6, 7}], 
 [{5, 6, 7}, {1, 2, 3, 8, 9, 10}, {1, 2, 3, 4, 5}], 
 [{6, 7, 8, 9, 10}, {1, 2, 3, 4, 5}]]

my expected output is

[[{1, 2, 3, 8, 9, 10}, {4, 5, 7}, {5, 6, 7}], 
 [{1, 2, 3, 4, 5}, {6, 7, 8, 9, 10}],  
 [{5, 6, 7}, {1, 2, 3, 8, 9, 10}, {1, 2, 3, 4, 5}]]

anyone knows any methods to solve the problem?

Does this answer your question? [Removing duplicates from a list of lists](https://stackoverflow.com/questions/2213923/removing-duplicates-from-a-list-of-lists) — Tom Myddeltyn, Dec 02 '20 at 23:14
@TomServo the problem with that is that list is not a hashable type. The other answer is probably better — L.Grozinger, Dec 02 '20 at 23:21
I have tried the ```itertools``` but it did not work out. The situation here is a little different because elements in the internal list are set instead of integers. — brillanting10, Dec 02 '20 at 23:22
Why do you have: [{5, 6, 7}, {1, 2, 3, 8, 9, 10}, {1, 2, 3, 4, 5}] in the output? — Dani Mesejo, Dec 02 '20 at 23:27
Take list [{5, 6, 7}, {1, 2, 3, 8, 9, 10}, {1, 2, 3, 4, 5}] as an example, three elements in this list {5, 6, 7}, {1, 2, 3, 8, 9, 10}, {1, 2, 3, 4, 5} can be used to cover the universe set = {1,2,3,4,5,6,7,8,9,10} — brillanting10, Dec 02 '20 at 23:32
for example [{1, 2, 3, 8, 9, 10}, {4, 5, 7}, {5, 6, 7}] and [{4, 5, 7}, {1, 2, 3, 8, 9, 10}, {5, 6, 7}] are two duplicates as well as [{1, 2, 3, 4, 5}, {6, 7, 8, 9, 10}] and [{6, 7, 8, 9, 10}, {1, 2, 3, 4, 5}] — brillanting10, Dec 02 '20 at 23:35

Dani Mesejo · Accepted Answer · 2020-12-02T23:45:00.157

You could do:

import pprint

data = [[{1, 2, 3, 8, 9, 10}, {4, 5, 7}, {5, 6, 7}],
        [{1, 2, 3, 4, 5}, {6, 7, 8, 9, 10}],
        [{4, 5, 7}, {1, 2, 3, 8, 9, 10}, {5, 6, 7}],
        [{5, 6, 7}, {1, 2, 3, 8, 9, 10}, {1, 2, 3, 4, 5}],
        [{6, 7, 8, 9, 10}, {1, 2, 3, 4, 5}]]

# find the uniques, keep order of appearance
uniques = dict.fromkeys([frozenset(frozenset(s) for s in e) for e in data])

# transform to original format
res = [[set(s) for s in e] for e in uniques]

pprint.pprint(res)

Output

[[{1, 2, 3, 8, 9, 10}, {5, 6, 7}, {4, 5, 7}],
 [{1, 2, 3, 4, 5}, {6, 7, 8, 9, 10}],
 [{1, 2, 3, 4, 5}, {1, 2, 3, 8, 9, 10}, {5, 6, 7}]]

The frozenset is a hashable version of a set. The function dict.fromkeys, keeps the order of appearance in Python 3.6+.

How to remove duplicates in a list containing lists of sets?

1 Answers1