Remove duplicates from list of lists

Question

I have a list containing ~300 lists, however some of them are duplicates and I would like to remove them. I've tried:

cleanlist = [cleanlist.append(x) for x in oldlist if x not in cleanlist]

but it keeps throwing RuntimeError: maximum recursion depth exceeded in comparison at me. I tried sys.setrecursionlimit(1500) but that didn't help.

What's a better way to do this?

How big are your lists? Are tuples maybe more appropriate? (They’re hashable, which makes getting a unique list – not in the right order, though – as easy as `list(set(oldlist))`.) — Ry-, Jun 04 '15 at 00:53
You're adding a bunch of stuff to `cleanlist` based on whether it's in `cleanlist` (which doesn't exist yet), and then saving the value returned from the `append` operation (which is `None`) to `cleanlist`. It will not end well. — TigerhawkT3, Jun 04 '15 at 00:53
Have a look at [How do you remove duplicates from a list in Python whilst preserving order?](http://stackoverflow.com/questions/480214/how-do-you-remove-duplicates-from-a-list-in-python-whilst-preserving-order) — Bhargav Rao, Jun 04 '15 at 00:56
@BhargavRao: Many of the answers there won’t work on a list of lists. — Ry-, Jun 04 '15 at 00:57

TigerhawkT3 · Answer 1 · 2015-06-04T00:58:23.313

6

You're adding a bunch of stuff to cleanlist based on whether it's in cleanlist (which doesn't exist yet), and then saving the value returned from the append() operation (which is None) to cleanlist. It will not end well.

A better way to do this is probably with an old-fashioned loop construct:

cleanlist = []
for x in oldlist:
    if x not in cleanlist:
        cleanlist.append(x)

edited Jun 04 '15 at 00:58

answered Jun 04 '15 at 00:55

TigerhawkT3

48,464
6
60
97

1

You could have added your comment to your answer :) It will be a better answer then! – Bhargav Rao Jun 04 '15 at 00:57
Good idea. I will do that. – TigerhawkT3 Jun 04 '15 at 00:57
I think `cleanlist` does already exist. Otherwise he'd get `NameError: name 'cleanlist' is not defined`, not `RuntimeError: maximum recursion...`. – Stefan Pochmann Jun 04 '15 at 01:09
1

If it exists, I don't know where or what it is. Either way, I'm not messing with Schrödinger's `list`. – TigerhawkT3 Jun 04 '15 at 02:33

João Paulo · Answer 2 · 2015-06-04T01:36:19.140

1

A crazy solution. Maybe It helps:

oldlist = [[1,2,3], [1,2,3], [4,5,6], [7,8], [7,8]]
cleanlist = set(tuple(x) for x in oldlist)
print(list(cleanlist))

>>> [(7, 8), (4, 5, 6), (1, 2, 3)]

For a list of lists, use this:

cleanlist = [list(item) for item in set(tuple(x) for x in oldlist)]
>>> [[7, 8], [4, 5, 6], [1, 2, 3]]

edited Jun 04 '15 at 01:36

answered Jun 04 '15 at 01:16

João Paulo

6,300
4
51
80

The result of this is a list of tuples, not a list of lists. – Paul Rooney Jun 04 '15 at 01:18
2

Or `list(map(list, set(map(tuple, oldlist))))`. – Stefan Pochmann Jun 04 '15 at 01:21
1

Depends. Some people might prefer the comprehension. The only really unclean thing in yours is that you create an unnecessary list that you give to `set`. – Stefan Pochmann Jun 04 '15 at 01:23
Oh, I overlooked the unnecessary set-to-list conversion, that's a second unclean thing. – Stefan Pochmann Jun 04 '15 at 01:26
this `list` at `set` can be cleaned. – João Paulo Jun 04 '15 at 01:28
1

You still create that pointless temporary list, `set(tuple(x) for x in oldlist)` would be better. – Stefan Pochmann Jun 04 '15 at 01:29
That's true! cleaned! – João Paulo Jun 04 '15 at 01:31

Remove duplicates from list of lists

2 Answers2