Eliminate duplicate from list of lists

Question

I need to eliminate duplicates from a list of list like this one:

    list = [[10, 5, 3], [10, 5, 3], [10, 10, 3], [10, 10], [3, 3, 3], [10, 5, 3]]

As a expected result:

    result_list = [[10, 5, 3], [10, 3], [10], [3]]

Eliminating duplicates inside sub-lists and in the main list, would it be possible? I tried with:

    result_list = [list(result) for result in set(set(item) for item in list)]

but throws an TypeError saying that a set is a unhashable type

I think it was not a duplicated question, i need to remove the duplicates within the sublists, not just in the main list. Thanks to everyone who helped me, problem solved.

Does order need to be preserved? What should happen to `[10, 3, 10]`? — Kelly Bundy, Feb 04 '20 at 14:46
maybe this is helpful https://www.w3resource.com/python-exercises/list/python-data-type-list-exercise-69.php — elmo26, Feb 04 '20 at 14:46

score 1 · Answer 1 · answered Feb 04 '20 at 14:44

Sets aren't hashable, but frozensets are:

lst = [[10, 5, 3], [10, 5, 3], [10, 10, 3], [10, 10], [3, 3, 3], [10, 5, 3]]
result_list = [list(result) for result in set(frozenset(item) for item in lst)]

Also don't shadow the builtin name list, especially if you want to use its usual meaning immediately after.

score 0 · Accepted Answer · answered Feb 04 '20 at 14:43

0

You should use map in order to convert to tuple

result = [list(set(i)) for i in set(map(tuple, mylist))]

Output

[[3], [10, 3, 5], [10, 3], [10]]

answered Feb 04 '20 at 14:43

Mihai Alexandru-Ionut

47,092
13
101
128

1

Thanks, really elegant solution. – delete Feb 04 '20 at 15:29
@delete, then you Could accept the answer in order to help other people – Mihai Alexandru-Ionut Feb 04 '20 at 16:00
Sorry, I didn't know how to do it, done now. – delete Feb 04 '20 at 16:09

score 0 · Answer 3 · answered Feb 04 '20 at 14:48

You need to use tuples to be able to set, however in a nested list comprehension, you can turn this tuple items back to lists:

list_example = [[10, 5, 3], [10, 5, 3], [10, 10, 3], [10, 10], [3, 3, 3], [10, 5, 3]]
output = [list(x) for x in (list(set([tuple(set(result)) for result in list(set(item) for item in list_example)])))]
print(output)

Output:

[[10, 3, 5], [10], [10, 3], [3]]

score 0 · Answer 4 · answered Feb 04 '20 at 14:48

You are iterating a list:

for item in list

putting the results into a set:

set(item)

then putting the set into a set:

set(set(item)

For anything to go into a set it has to be hashable, meaning it has a defined hash value resulting from being an immutable object. sets aren't immutable, and so don't have a hash. See Why aren't Python sets hashable?.

Eliminate duplicate from list of lists

4 Answers4