3

Is there a way to remove duplicate sublists from a list of lists, even if they are not the same ordering?

So could I do something like make:

x = [[1,2],[3,4],[5,6],[2,1],[7,8]]

into

x = [[1,2],[3,4],[5,6],[7,8]]

Is there an itertools function or something with a for loop?

Thanks!

5813
  • 1,073
  • 3
  • 14
  • 28
  • Will you ever have to distinguish `[1,1,2]` from `[1,2,1]`? And do you need to preserve the initial order of the sublists? – DSM Dec 09 '13 at 03:40
  • 2
    Also, is the order of the returned list significant? – Burhan Khalid Dec 09 '13 at 03:40
  • @DSM yes, the length of the sublists may increase. And no, the order is not important. – 5813 Dec 09 '13 at 03:44
  • @DSM, but the order within the sublists is important, so [7,8] is not the same as [8,7]. – 5813 Dec 09 '13 at 03:50
  • @DSM Yes, that is what I mean. If for a sublist there is a matching one elsewhere in the list, then one, preferably the second, should be removed. But, for a sublist which does not have a duplicate, it should remain the same. – 5813 Dec 09 '13 at 03:55

3 Answers3

4

this will preserve the order of list and sublists, with possible duplicates in sublists:

y, s = [], set()
for t in x:
    w = tuple(sorted(t))
    if not w in s:
        y.append(t)
        s.add(w)

if

x = [[1,2],[3,4],[5,6],[2,1,1],[2,1],[7,8],[4,3],[1,2,1]]

then y will be:

[[1, 2], [3, 4], [5, 6], [2, 1, 1], [7, 8]]
behzad.nouri
  • 74,723
  • 18
  • 126
  • 124
  • 1
    +1: This preserves as much original information as possible (unlike my OrderedDict approach), and seems to satisfy the OP's desiderata. – DSM Dec 09 '13 at 03:55
1

You can use frozenset: 

>>> def remove_dups(L):
        return map(list, frozenset(map(frozenset, L)))

>>> x = [[1,2],[3,4],[5,6],[2,1],[7,8]]
>>> remove_dups(x)
[[5, 6], [1, 2], [8, 7], [3, 4]]
>>>
0

try this:

a=[[1,2],[3,4],[5,6],[2,1],[7,8]]
y=[]
for i in a:
        if sorted(i) not in y:
                y.append(i)
print y

output is

[[1, 2], [3, 4], [5, 6], [7, 8]]
Ashif Abdulrahman
  • 2,077
  • 1
  • 10
  • 4