How to remove duplicate lists in a list of list?

Question

I want to remove all duplicates list from a list of list.

So I have a list of lists like this.

a = [[1,2],[1,2],[3,4,5],[3,4,5],[3,4,5]]

I want to have:

b = [[1,2],[3,4,5]]

I don't know how to do it.

score 81 · Accepted Answer · edited Jun 21 '23 at 19:30

81

You could use a set:

b_set = set(map(tuple,a))  #need to convert the inner lists to tuples so they are hashable
b = list(map(list,b_set)) #Now convert tuples back into lists (maybe unnecessary?)

Or, if you prefer list comprehensions/generators:

b_set = set(tuple(x) for x in a)
b = [ list(x) for x in b_set ]

Finally, if order is important, you can always sort b:

b.sort(key = lambda x: a.index(x) )

edited Jun 21 '23 at 19:30

Sadra

2,480
2
20
32

answered Aug 30 '12 at 13:52

mgilson

300,191
65
633
696

1

If you want something to be unique, a set is the way to go. :) – mgilson Aug 30 '12 at 13:54
`lambda x: a.index(x)` could be just `a.index` – Jean-François Fabre Oct 26 '18 at 19:01
2

`b = map(list,b_set)` will return a map of lists, if you want it back to the original list of lists you'll want to instead do `b = list(map(list,b_set))` – defermat Apr 19 '19 at 18:01

score 15 · Answer 2 · answered Aug 30 '12 at 13:54

15

See mgilson's answer if the order of the lists is not important. If you want to retain the order, do something like:

b = list()
for sublist in a:
    if sublist not in b:
        b.append(sublist)

This will keep the order in the original list. However, it is slower and more verbose than using sets.

answered Aug 30 '12 at 13:54

HerrKaputt

2,594
18
17

Thank you too, your method is interesting, but I don't need to keep the order. – user1507156 Aug 30 '12 at 14:12
Just note that this method will treat [1,2] and [2,1] as different, though it works for this example. – Zoe L Aug 01 '18 at 23:28
This method is slow. Avoid using append – Shekhar Samanta Sep 28 '21 at 04:26

How to remove duplicate lists in a list of list?

2 Answers2

Linked

Related