Remove both duplicates in multiple lists python

Question

I have three lists X, Y, Z as follows:

X: [1, 1, 2, 3, 4, 5, 5, 5]
Y: [3, 3, 2, 6, 7, 1, 1, 2]
Z: [0, 0, 1, 1, 2, 3, 3, 4]

I am trying to remove both duplicated set of values at the same index of the lists get a reduced list as follows, all three list will always have the same length initially and at the end as well:

X: [2, 3, 4, 5]
Y: [2, 6, 7, 2]
Z: [1, 1, 2, 4]

I tried using the zip(X, Y, Z) function but I can't index it and the dict.fromkeys only removes one of the duplicates and leaves the other in the new list. I want to be able to remove both.

Any help is appreciated!

I don't understand your logic. Why does the third list still have duplicates? It also seems like you are removing arbitrary values. — user3483203, Jun 13 '18 at 14:27
More importantly, this question is missing a [mcve], as well as your attempt to solve your own problem. — user3483203, Jun 13 '18 at 14:28
Ah I understand now. He wants to remove duplicates there there is a pair of duplicate values at the same two indices in each of the three lists. So here indicies 0-1 are removed, and 5-6 are removed. Very unclear question, but now I get the logic. — user3483203, Jun 13 '18 at 14:30

jpp · Accepted Answer · 2018-06-13T14:43:29.663

5

Using collections.Counter and zip, you can count unique triplets.

Then remove duplicates via a generator comprehension.

from collections import Counter

X = [1, 1, 2, 3, 4, 5, 5, 5]
Y = [3, 3, 2, 6, 7, 1, 1, 2]
Z = [0, 0, 1, 1, 2, 3, 3, 4]

c = Counter(zip(X, Y, Z))

X, Y, Z = zip(*(k for k, v in c.items() if v == 1))

print(X, Y, Z, sep='\n')

(2, 3, 4, 5)
(2, 6, 7, 2)
(1, 1, 2, 4)

Note if ordering is a requirement and you are not using Python 3.6+, you can create an "OrderedCounter" instead by subclassing collections.OrderedDict.

edited Jun 13 '18 at 14:43

answered Jun 13 '18 at 14:31

jpp

159,742
34
281
339

1

correct me if I'm wrong, but this does not guarantee that the order of the triplets will be preserved - albeit, the triplets themselves will be. I'm not the down-voter, just an observation – Jared Goguen Jun 13 '18 at 14:39
1

@JaredGoguen, Good point: in Python 3.6+ it would, not in <3.6. You could construct an OrderedCounter (subclassing `collections.OrderedDict`), but not sure if order is a requirement and OP's Python version. – jpp Jun 13 '18 at 14:41
1

that is a side effect of the [CPython implementation](https://stackoverflow.com/questions/39980323/are-dictionaries-ordered-in-python-3-6) and is not, in general, guaranteed in 3.6. It will however be guaranteed in 3.7 – Jared Goguen Jun 13 '18 at 14:42
1

of course, we don't even know if preserving order is required by the OP – Jared Goguen Jun 13 '18 at 14:44

koPytok · Answer 2 · 2018-06-13T14:37:19.403

1

It's convenient to use pandas library for the task. Just create dataframe using the lists and apply df.drop_duplicates with keep=False (means remove all duplicated rows):

import pandas as pd

dct = {
"X": [1, 1, 2, 3, 4, 5, 5, 5],
"Y": [3, 3, 2, 6, 7, 1, 1, 2],
"Z": [0, 0, 1, 1, 2, 3, 3, 4],
}
d = pd.DataFrame(dct)
d.drop_duplicates(keep=False)

edited Jun 13 '18 at 14:37

answered Jun 13 '18 at 14:31

koPytok

3,453
1
14
29

score 0 · Answer 3 · answered Jun 13 '18 at 14:38

Not the best possible approach

>>> from collections import Counter
>>> zipped_items = list(zip(x,y,z))
>>> counts = Counter(zipped_items)
>>> filtered_items = [item for item in zipped_items if counts[item] == 1]
>>> x1, y1, z1 = [ list(map(lambda x: x[i], filtered_items))
... for i in range(3)]

score 0 · Answer 4 · answered Jun 13 '18 at 15:20

Here is my solution without any import, but still short and easily readable:

X = [1, 1, 2, 3, 4, 5, 5, 5]
Y = [3, 3, 2, 6, 7, 1, 1, 2]
Z = [0, 0, 1, 1, 2, 3, 3, 4]

zipped = list(zip(X, Y, Z))
X, Y, Z = zip(*[i for i in zipped if zipped.count(i) == 1])
X, Y, Z = list(X), list(Y), list(Z)    

print(X, Y, Z, sep='\n')
# [2, 3, 4, 5]
# [2, 6, 7, 2]
# [1, 1, 2, 4]

Remove both duplicates in multiple lists python

4 Answers4

Linked