How to remove duplicates from nested lists

Question

I have a lists as list<list<list>> lists where I want to remove duplicate list.

list_1 = [
           [
             [1,2],[2,3]
           ],
           [
             [3,4],[5,6]
           ],
           [
             [3,4], [5,6]
           ]

         ]

So expected output should be

output = [
           [
             [1,2],[2,3]
           ],
           [
             [3,4],[5,6]
           ]
         ]

Is there are shortcut to do that instead of comparing each list with each other. We cannot do set(list_1), then what is the easiest to remove duplicates?

P.S: It's a 3 level nesting so the answer marked for duplicate question doesn't work.

@pistol2myhead It only works for 2 level nesting, I have 3 level nesting. — Atihska, Mar 26 '19 at 04:51

score 3 · Accepted Answer · answered Mar 26 '19 at 05:09

You can try this

k = [
           [
             [1,2],[2,3]
           ],
           [
             [3,4],[5,6]
           ],
           [
             [3,4], [5,6]
           ]

         ]

import itertools
k.sort()
list(k for k,_ in itertools.groupby(k))
[[[1, 2], [2, 3]], [[3, 4], [5, 6]]]

For detailed info you can refer here https://stackoverflow.com/a/2213973/4320263

DirtyBit · Answer 2 · 2019-03-26T05:39:03.183

One way I see to it, flatten the list to a lower level and then make this lazy, and at the same time preserve order, by iterating the list of lists and adding to a "seen" set:

lst = [ [[1,2],[2,3]], [[3,4],[5,6]], [[3,4], [5,6]] ]

from toolz import unique

flat_list = [item for sublist in lst for item in sublist]
res = map(list, unique(map(tuple, flat_list)))
print(list(res))

OUTPUT:

[[1, 2], [2, 3], [3, 4], [5, 6]]

EDIT:

And if you want the nested lists back, pairing every 2 elements:

print([res[i:i+2] for i in range(0, len(res), 2)])

OUTPUT:

[[[1, 2], [2, 3], [3, 4]], [[5, 6]]]

How to remove duplicates from nested lists

2 Answers2

Linked