0

Trying to remove duplicates in list of list and storing it.

Original List:

list1 = [[a, b, a, b], [b, c, d, c], [a, c, c]]

Looking for output:

list1 = [[a, b], [b,c,d], [a,c]]

My code:

unique_list = []
for sublist in list1:
    element = [elem for elem in sublist if elem not in unique_list]
    if len(element):
        unique_testset.append(element)

My code appends a sublist multiple times and doesn't get rid of the duplicates.

Miztory
  • 158
  • 1
  • 9
  • https://stackoverflow.com/questions/2213923/removing-duplicates-from-a-list-of-lists – OmegaOdie Sep 18 '20 at 10:40
  • 3
    Does this answer your question? https://stackoverflow.com/questions/7961363/removing-duplicates-in-lists `without_duplicates = [list(set(sublist)) for sublist in list1]` – Stef Sep 18 '20 at 10:41
  • @OmegaOdie There is a confusion here; I believe the OP is trying to remove duplicate elements in each sublist; the question you linked is trying to remove duplicate sublists. – Stef Sep 18 '20 at 10:42
  • Here your answer https://stackoverflow.com/questions/55349986/how-to-remove-duplicates-from-nested-lists – Muhammad Rizwan Munawar Sep 18 '20 at 10:43

4 Answers4

3

Removing duplicates the simple way

A classic, efficient way to remove duplicates from a list in python is to build a set from the list: removing duplicates in lists

list_with_dups = [1, 1, 2, 3, 2]
list_without_dups = list(set(list_with_dups))

You can apply this method repeatedly using a list comprehension:

list1 = [['a', 'b', 'a', 'b'], ['b', 'c', 'd', 'c'], ['a', 'c', 'c']]
without_duplicates = [list(set(sublist)) for sublist in list1]
#                  = [['b', 'a'], ['d', 'b', 'c'], ['c', 'a']]

Removing duplicates whilst conserving order

Applying How do you remove duplicates whilst conserving order? to a list of lists:

def f7(seq):
    seen = set()
    seen_add = seen.add
    return [x for x in seq if not (x in seen or seen_add(x))]

list1 = [['a', 'b', 'a', 'b'], ['b', 'c', 'd', 'c'], ['a', 'c', 'c']]
without_duplicates = [f7(sublist) for sublist in list1]
#                  = [['a', 'b'], ['b', 'c', 'd'], ['a', 'c']]
Stef
  • 13,242
  • 2
  • 17
  • 28
  • Dear @MuhammadRizwanMunawar, thank you for trying to improve my answer by suggesting an edit. Your edit suggestion to my answer is actually just deleting parts of my answer rather than improving it. This is doubly annoying because: 1) I cannot reject your edit. 2) I wanted to edit my answer to make it clearer, but I cannot, because your suggested edit prevents me from editing further. Please revert your edit. I do not understand why you decided to delete parts of my answer and this is very annoying. – Stef Sep 18 '20 at 10:54
  • Thanks! This works for me but it changes the order of the sublist. Is there any way to preserve the order? – Miztory Sep 18 '20 at 10:54
  • @Miztory Yes, `set` structures in python do not conserve order. Here is a stackoverflow question about removing duplicates whilst conserving order: https://stackoverflow.com/questions/480214/how-do-you-remove-duplicates-from-a-list-whilst-preserving-order – Stef Sep 18 '20 at 10:57
  • @Stef I think I'll be able to continue with my work even with a different order.Thanks for the help. :) – Miztory Sep 18 '20 at 11:13
  • @Miztory I've updated the answer to conserve order. – Stef Oct 01 '20 at 10:28
0

try this one...

  list1 = [[a, b, a, b], [b, c, d, c], [a, c, c]]
    
  import itertools
    list1.sort()
    list(list1 for list1,_ in itertools.groupby(list1))
Muhammad Rizwan Munawar
  • 672
  • 4
  • 14
0

One way you can achieve this is using:

list1 = [[a, b, a, b], [b, c, d, c], [a, c, c]]

for sublist in list1:
   l = list(set(sublist))
   if l:
     unique_list.append(l)

set removes all the duplicate values and we have unique values

bhavin tandel
  • 65
  • 10
0

Use This code it'll help you

    import numpy as np
    lis=[['a','b','a'], ['b','c','d'], ['c','a','c']]
    uni=np.unique(lis)
    print(uni)
Vinesh
  • 1