Python: Randomly remove duplicates among multiple lists

Question

Given

x = [0,1,2,3]
y = [0,4,3,5]
z = [1,6,7,8]

I need something that removes all duplicates over all three lists randomly. So the desired outcome would be something like:

x = [1,2,3]
y = [0,4,5]
z = [6,7,8]

Running it another time should give a different distribution like for example:

x = [0,1,2,3]
y = [4,5]
z = [6,7,8]

All questions I've already found (like this one for example Remove both duplicates in multiple lists python) remove duplicates per list.

Thanks a lot in advance!

EDIT: Here is my current fruitless effort:

duplicatelist = []
    for number in y:
        if number in x:
            duplicatelist.append(number)
    for number in z:
        if number in x or number in y:
            duplicatelist.append(number)

#now I don*t know how to continue

I've run my head against the issue multiple times. I was thinking looping through all of them and then adding the duplicates to a fourth list or dictionary but I didn't know a good way to use that fourth list to remove the duplicates from the first three. — Difio, Oct 25 '22 at 07:38
What are the criteria of removing duplicates? Should one of them always be removed? What if there's 4 or more equal items? — Nikolaj Š., Oct 25 '22 at 10:55
After the operation there should be no duplicates left. The rest should be completely random. — Difio, Oct 25 '22 at 11:02

score 1 · Accepted Answer · edited Oct 25 '22 at 10:49

1

If you start randomly (means x,y, z which one to treat first) you could get what you want.

import random
def rm_dup(**kwargs):
    inp = kwargs
    duplicate = set()
    keys = list(inp.keys())
    random.shuffle(keys)
    out = {k: [] for k in inp.keys()}

    for key in keys:
        for v in inp[key]:
            if v not in duplicate:
                out[key].append(v)
                duplicate.add(v)
    return out

example outs:

out = rm_dup(x=x,y=y,z=z)
print(out)

{'x': [2], 'y': [0, 4, 3, 5], 'z': [1, 6, 7, 8]}

{'x': [0, 1, 2, 3], 'y': [4, 5], 'z': [6, 7, 8]} ...

edited Oct 25 '22 at 10:49

Thierry Lathuille

23,663
10
44
50

answered Oct 25 '22 at 08:20

amirhm

1,239
9
12

Hi amirhm! I'll only get to try it this evening but thanks a lot already for your suggestion. I'll let you know if it works for me. – Difio Oct 25 '22 at 09:21
Hi Amirhm! Thanks so much! It works beautifully! – Difio Oct 25 '22 at 10:51
@Difio You are very welcome. I believe you can mark it as answer accepted. – amirhm Oct 25 '22 at 19:28

Python: Randomly remove duplicates among multiple lists

1 Answers1