0

I have a list of lists in Python and want to remove duplicates with each element being compared and yes, I care (the most) about the order of the list.

I have already looked at most of the solutions on stack overflow. For e.g. the one here, solution voted doesn't compare the elements of list of lists and would fine for the given use case, provided it's sorted. However, what I'm looking for is:

What's available

s1 = ['Lucifer', 'Ella']
s2 = ['Lucifer', 'Eve']
s3 = ['Chloe', 'Lucifer']
s4 = ['Lucifer', 'Linda']

What's required

As I said, care about the order, remove duplicate from each list (that's why 'Lucifer' is missing from the list) and get only unique ones..

['Ella', 'Eve', 'Chloe', 'Linda']

Caution: It will not always be the first element that would be common across list for e.g. in s4 it can be 'Amenadiel'. For now let's assume that length of lists to be compared is always 2.

What have I tried

list((set(s1) ^ set(s2) ^ set(s3) ^ set(s4)) ^ (set(s1) & set(s2) & set(s3) & set(s4)))

the output from above is ['Chloe', 'Eve', 'Linda', 'Ella', 'Lucifer'] which is not as expected (as mentioned above!).

This was working fine with two or three list, but isn't working with four list.

Please help!

TIA.

PratikSharma
  • 321
  • 2
  • 17

3 Answers3

3

You can first find the duplicates with a Counter, then filter them out in a second pass:

from collections import Counter

s1 = ['Lucifer', 'Ella']
s2 = ['Lucifer', 'Eve']
s3 = ['Chloe', 'Lucifer']
s4 = ['Lucifer', 'Linda']
lists = [s1, s2, s3, s4]
flattened = [item for sublist in lists for item in sublist]

counts = Counter(flattened)
deduped = [x for x in flattened if counts[x] <= 1]

print(deduped) # ['Ella', 'Eve', 'Chloe', 'Linda']

If your goal is to only remove elements in every list:

s1 = ['Lucifer', 'Ella']
s2 = ['Lucifer', 'Eve']
s3 = ['Chloe', 'Lucifer']
s4 = ['Lucifer', 'Linda']
lists = [s1, s2, s3, s4]
flattened = [item for sublist in lists for item in sublist]

seen = set.intersection(*map(set, lists))
deduped = []
for item in flattened:
    if item in seen:
        continue
    seen.add(item)
    deduped.append(item)
print(deduped) # ['Ella', 'Eve', 'Chloe', 'Linda']
Aplet123
  • 33,825
  • 1
  • 29
  • 55
  • If you're using `collections`, then maybe a prettier way to flatten lists would be to use `chain(lists)` – go2nirvana Nov 20 '20 at 15:16
  • +1 upvote, works well. But as I said in the caution point, if 'Lucifer' in s4 is replaced by 'Amenadiel', it would add him in the output but leave out Lucifer, count of which would be 3. – PratikSharma Nov 20 '20 at 15:45
  • 1
    Is that not the behavior you want? – Aplet123 Nov 20 '20 at 15:48
  • If I was to replace 'Lucifer' with 'Amenadiel' the expected output would be ```['Lucifer', 'Ella', 'Eve', 'Chloe', 'Amenadiel', 'Linda']```. – PratikSharma Nov 20 '20 at 16:09
  • ...but Lucifer is a duplicate so it should be removed, right? Unless you're only supposed to remove elements if they appear in every list? – Aplet123 Nov 20 '20 at 16:10
  • yes, only if it appears in every list, remove that and keep only unique of the ones which doesn't appear in every list. – PratikSharma Nov 20 '20 at 16:12
  • 1
    @PratikSharma I've edited my answer to include a working solution. – Aplet123 Nov 20 '20 at 16:18
2

An alternative solution using also Counter and chain:

from collections import Counter
from itertools import chain

s1 = ['Lucifer', 'Ella']
s2 = ['Lucifer', 'Eve']
s3 = ['Chloe', 'Lucifer']
s4 = ['Lucifer', 'Linda']

# count the occurrences
counts = Counter(chain(s1, s2, s3, s4))

# remove duplicates
result = [name for name in chain(s1, s2, s3, s4) if counts[name] < 2]

print(result)

Output

['Ella', 'Eve', 'Chloe', 'Linda']

UPDATE

If you want those that do not appear in all the lists, do:

from collections import Counter, OrderedDict
from itertools import chain

s1 = ['Lucifer', 'Ella']
s2 = ['Lucifer', 'Eve']
s3 = ['Chloe', 'Lucifer']
s4 = ['Amenadiel', 'Linda']

all_ls = [s1, s2, s3, s4]

# count the occurrences
counts = Counter(chain.from_iterable(all_ls))

# remove duplicates i.e do not appear in all lists
names = list(OrderedDict.fromkeys(chain.from_iterable(all_ls)))
result = [name for name in names if counts[name] < len(all_ls)]

print(result)

Output

['Lucifer', 'Ella', 'Eve', 'Chloe', 'Amenadiel', 'Linda']
Dani Mesejo
  • 61,499
  • 6
  • 49
  • 76
0

One way to do this is to add up all the lists initially. Then, convert this list into a dictionary. Thereafter, convert this temporary dictionary into a list.

s1 = ['Lucifer', 'Ella']
s2 = ['Lucifer', 'Eve']
s3 = ['Chloe', 'Lucifer']
s4 = ['Lucifer', 'Linda']

s_all = s1 + s2 + s3 + s4       # Concatenate all lists

temp_dict = dict.fromkeys(s)    # Create a dictionary with keys from list `s_all`
s_no_dupes = list(temp_dict)    # Convert the dictionary to a list

print(s_no_dumpes)

Output

['Lucifer', 'Ella', 'Eve', 'Chloe', 'Linda']

Note: This will remove all the duplicates but if you want them to be in order, you must have Python 3.6 or above, since, it was from Python 3.6 that the dictionary keys were ordered.

Melvin Abraham
  • 2,870
  • 5
  • 19
  • 33