How to remove duplicate lists from a 'list of lists'?

Question

I have a list of lists (I'm relatively new to Python so excuse me if the terms are inaccurate, but look at the example below) and want to remove any duplicate lists.

In this example, entries 1&4 and 3&5 are identical and a duplicate should be removed.

List = [[1, 'A', 6, 2], [8, 'C', 6, 2], [3, 'G', 3, 4], [1, 'A', 6, 2], [3, 'G', 3, 4], [3, 'B', 3, 4]]

[[1, 'A', 6, 2], [8, 'C', 6, 2], [3, 'G', 3, 4], [3, 'B', 3, 4]]

I currently have the following for loop reading through the list and removing duplicates but this makes it very slow and my code is much longer and the input list is much more complicated than in my example and makes the code run for days and days.

unique = []
for i in cohesiveFaceNodes:
    if not i in unique:
        unique.append(i)
cohesiveFaceNodes = unique

Do you need to preserve the order? Does each element in the resulting list have to be a list (say, rather than a tuple)? — j1-lee, Aug 23 '22 at 18:10
I think this post can help you https://stackoverflow.com/questions/2213923/removing-duplicates-from-a-list-of-lists — Nexo, Aug 23 '22 at 18:13
`unique = list(map(list, dict.fromkeys(map(tuple, arr))))` preserves types and order. — Steven Rumbalski, Aug 23 '22 at 18:27

score 3 · Answer 1 · answered Aug 23 '22 at 18:21

de-duping while preserving the order (from Cpython 3.6+):

>>> lst = [[1, 'A', 6, 2], [8, 'C', 6, 2], [3, 'G', 3, 4], 
...        [1, 'A', 6, 2], [3, 'G', 3, 4], [3, 'B', 3, 4]]
>>> [list(x) for x in dict.fromkeys(map(tuple, lst))]
[[1, 'A', 6, 2], [8, 'C', 6, 2], [3, 'G', 3, 4], [3, 'B', 3, 4]]

JRiggles · Answer 2 · 2022-08-23T18:34:26.470

2

If you can convert the inner lists into tuples, there is a super simple one-liner way to handle this

# use a list of tuples instead of a list of lists for this method to work
input_list = [(1, 'A', 6, 2), (8, 'C', 6, 2), (3, 'G', 3, 4), (1, 'A', 6, 2), (3, 'G', 3, 4), (3, 'B', 3, 4)]
deduped_list = list(dict.fromkeys(input_list))  # remove dupes, return new list of tuples

Edit to add that a quick way to convert your existing list of lists to a list of tuples is to use a list comprehension like so input_list = [tuple(e) for e in input_list]

Edit 2: if you for some reason really really need a list of lists after the fact, once again it's list comprehensions to the rescue final_list = [list(e) for e in deduped_list]

edited Aug 23 '22 at 18:34

answered Aug 23 '22 at 18:20

JRiggles

4,847
1
12
27

1

to preserve order do `deduped_list = list(dict.fromkeys(inputlist))` – Steven Rumbalski Aug 23 '22 at 18:25
@StevenRumbalski good call! Edited – JRiggles Aug 23 '22 at 18:28
you seem to have edited your answer into mine ;) – Chris_Rands Aug 23 '22 at 19:54
@Chris_Rands if it ain't broke...? hahaha – JRiggles Aug 24 '22 at 13:31

score 0 · Answer 3 · answered Aug 23 '22 at 18:18

Testing whether something is an element of a list (i in unique) is quite expensive (it iterates the list element by element until it finds a match or the list is exhausted). To check for element membership a data structure such as a set is much more efficient. So making unique a set rather than a list would help.

Now there's a small hurdle: Python sets don't support lists as members, because lists are mutable and not hashable. Assuming the elements in each of the inner lists are hashable, though, you can convert them to Python tuples (which are similar to lists but immutable) and then they can be elements of sets.

So one solution could be (I'm reusing the original variable names, though I think they're not ideal and I recommend changing them):

unique = set()
result = []
for i in cohesiveFaceNodes:
    i_as_tuple = tuple(i)
    if not i_as_tuple in unique:
        unique.add(i_as_tuple)
        result.append(i)

score 0 · Answer 4 · edited Aug 24 '22 at 15:23

For better coding practice and readability, it may be better to use dataclass to store these data. You can explicitly name each entry in the inner list for more clarity. dataclass offers built-in equality comparison like the tuple methods in the other answers.

from dataclasses import dataclass

@dataclass(unsafe_hash=True)
class AClass:
    some_int: int
    some_chr: str
    int2: int
    int3: int


lst = [[1, 'A', 6, 2], [8, 'C', 6, 2], [3, 'G', 3, 4], 
       [1, 'A', 6, 2], [3, 'G', 3, 4], [3, 'B', 3, 4]]

new_lst = [AClass(*x) for x in lst]
deduped_list = list(dict.fromkeys(new_lst))

How to remove duplicate lists from a 'list of lists'?

4 Answers4